Scenario Depot Blues [Archive]

Admiral Keth

10-22-2004, 10:41 AM

To paraphrase the famous quote, "You can't please everyone all the time".

This has been covered in threads long gone, but the predominant reason the old cumulative rating system was scrapped was that it was supremely subject to "ballot box stuffing", abuse, and whining. Players were intentionally submitting either overly high or overly low reviews to skew the cumulative score. Authors were perpetually complaining that players were posting low review scores which adversely affected their overall high rating.

This made the entire cumulative rating system invalid. Now each review stands on its own merits, and contributes nothing to an overall score.

If anyone can design a system where players can post scores which accumulate to an overall rating, which also cannot be maliciously manipulated, I'm open to ideas. I'm skeptical that it can be built, simply due to the fact that if a player truly hated a scenario and posted a low score which devalued the overall score, the author will claim "trolling" and petition to have the review removed.

Prior to The Scenario Depot v2 redesign, I petitioned for suggestions from players and authors on what features they would like to see implemented. Pretty much everything that was capable of being coded was implemented.

Maybe now is the time to begin taking in ideas on what people would like to see in The Scenario Depot v3. If you preferred the old system, then help design the new system. If you want your suggestion to have any merit, you must consider the following:

* How to prevent abuse by both players (reviewers) and authors.

* What to do with the reviews from the old systems. Incorporate or segregate?

* How to rank various scenarios against each other. For example, should historical scenarios be ranked in the same categories as fictional?

* Now that authors can define their own rating labels, how will those scenarios be ranked? Perhaps have 2 to 4 hard-coded categories, such as Play vs. AI, then allow perhaps 2 to 4 additional author-defined categories.

* Authors may selectively choose to have numeric, textual, both numeric and textual, or no reviews at all. What to do about scenarios which have no reviews, or simply textual reviews? Perhaps how the previous problem is handled will define how this one is solved.

* A hard-and-fast set of rules by which players and authors alike agree that reviews can be stricken from the overall score. Perhaps something similar to a flag which can be set by the author, which publicly identifies the review as a possible troll-attempt, allows the poster to respond, and removes all values from the cumulative score. Hmmm, actually not a bad idea.

I have no problem re-instituting the previous system, but prior to doing so, there MUST be a viable solution to the previous problems.

Now is the time for everyone to weigh in on this. I'll potentially be out of work in February, so this is a good time to begin putting together the specifications for this. No idea will be refused, but those ideas which actual include some methodology (coding, math, logic) will be given more weight.

=====================
@jwxspoon - Please clarify your "1 for 1" idea. How can that be enforced?

Kingfish

10-22-2004, 10:56 AM

Originally posted by Admiral Keth:
=====================
@jwxspoon - Please clarify your "1 for 1" idea. How can that be enforced? The 'Enforcer' will make sure you get your "1 for 1"

http://www.geocities.com/jwxspoon/jaegersized.JPG

Richie

10-22-2004, 11:15 AM

Originally posted by Admiral Keth:
To paraphrase the famous quote, "You can't please everyone all the time".

* Authors may selectively choose to have numeric, textual, both numeric and textual, or no reviews at all.

* A hard-and-fast set of rules by which players and authors alike agree that reviews can be stricken from the overall score. Perhaps something similar to a flag which can be set by the author, which publicly identifies the review as a possible troll-attempt, allows the poster to respond, and removes all values from the cumulative score. Hmmm, actually not a bad idea.

True, and both good ideas!

If a score seems inconsistent and a designer wishes the score to be striken from the record (but a review should always stand, as trolling is usually pretty obvious and will weigh against a reviewer) then let it be noted.

Perhaps you could have a continuous scoring system for the reviewer. A conglomerate score that represents their reviewing history, much like with ebay's feedback with the number of scenarios reviewed and an agregate score...

Hmmm....

or we could just have the 'ENFORCER' to deal with the trolls

Admiral Keth

10-22-2004, 11:25 AM

Originally posted by Richie:
<blockquote>quote:<hr />Originally posted by Admiral Keth:
* Authors may selectively choose to have numeric, textual, both numeric and textual, or no reviews at all.

...and both good ideas!
[/QUOTE]Not sure what you meant, but this is already an implemented feature for the authors. There are default values which reflect the old rating labels, but authors may change them.

Originally posted by Richie:

If a score seems inconsistent and a designer wishes the score to be striken from the record (but a review should always stand, as trolling is usually pretty obvious and will weigh against a reviewer) then let it be noted.

Perhaps you could have a continuous scoring system for the reviewer. A conglomerate score that represents their reviewing history, much like with ebay's feedback with the number of scenarios reviewed and an agregate score...

Hmmm.... Let me see if I follow you correctly...

A link (perhaps tied to the reviewer's name), which displays a list of reviews posted by that reviewers, perhaps with their respective scores. Thus, if for some reason the reviewer posted 1's in a particular scenario, a reader or author can determine if the reviewer has a history of posting low scores, then discount their review. Until the author flagged the review as a troll-attempt, it would be incumbent upon the reader to mentally deduct the reviewer's score from the cumulative score.

Admiral Keth

10-22-2004, 11:35 AM

Originally posted by Richie:
...or we could just have the 'ENFORCER' to deal with the trolls Although a machine gun may be a bit much, perhaps a dedicated team of "Review Reviewers" might keep things in line.

Steve Overton and his group have been instrumental and extremely helpful in several areas, including patrolling The Scenario Depot for various reasons, but I'd rather they spent the bulk of their effort making more scenarios.

I'm now envisioning two seperate flags for reviews: troll and hidden. The Troll flag discounts the associated rating from the cumulative score, and emails the reviewer that the author has flagged the review, but the review still remains publicly visible. The Hidden flag removes the review ratings from the cumulative score, hides the review, and emails the reviewer.

At such time that the new system is implemented, I can forsee one or two "Moderator" positions occurring, which would have the capability of touching reviews, and flagging them as Troll or Hidden, along with emailing the reviewer. The email FROM address sent to the reviewer will be that of the person doing the flagging, so that the reviewer can respond to the correct person.

Sergei

10-22-2004, 11:41 AM

How much is 'trolling' a real problem, and how much is it of that people have very different mind sets, experiences etc.? I think there are people who give scenarios weird ratings with odd explanation, but I don't think all of them really are trolls. Just... different.

Maybe you could have the possibility of rating ratings, maybe, in a "was this review useful?" way. If a review has a certain amount of negative ratings, then it would have diminished or no weight in the overall score. Designer couldn't vote (but he can of course leave his response and ask to explain if the review was strange) to avoid him from purposely downgrading the value of negative reviews no matter how reasoned the opinion...

It sounds unnecessarily complex, though. And then of course you'll have to develop measures to prevent trolling the review rating system... :D :D :D

Richie

10-22-2004, 11:51 AM

OK, forgive me, I've had issues uploading to The Depot in the past but that was entirely my own making and no reflection on the excellent website and resource you run. Please don't get me wrong, I'm trying to be constructive.

My inexperience is my own fault...

a) This is a feature available on initial upload or a feature available at any time with the current version?

For example, the 2 reviews I have for my Reichstag battle can be deleted or made concurrent?

b) Yes, the reviewer grades, for example, 3 scenarios at 1.5, 5 and 3.5 for a cummulative reviewer score of 3.3... a bit like I have 355 posts, at the mo'.

c) I think a review should stand regardless but with no score. Flagged as score removed by the designer. With an average reviewer score and the review itself, readers should be able to detect trolling, but a moderator would certainly help.
Now there's alot of commitment!

[ October 22, 2004, 09:03 AM: Message edited by: Richie ]

jwxspoon

10-22-2004, 11:52 AM

Admiral - what would be neat wouuld be to have a "point system" awarded for scenario reviews. A new registrant at the Scenario Depot might start with 10 points in his 'account'. As he downloads new scenarios, his account balance is reduced. When he posts reviews, his acount balance is credited. Etc.

Kingfish - You bastard. Why did I ever post some of those old army pics?

jw

Admiral Keth

10-22-2004, 12:02 PM

Originally posted by Richie:
a) This is a feature available on initial upload or a feature available at any time with the current version?
The author may change the labels associated with the numeric values at any time.

Originally posted by Richie:

For example, the 2 reviews I have for my Reichstag battle can be deleted or made concurrent?
This confused me.

Originally posted by Richie:

b) Yes, the reviewer grades, for example, 3 scenarios at 1.5, 5 and 3.5 for a cummulative reviewer score of 3.3... a bit like I have 355 posts, at the mo'.
OK, that appears to be a viable method by which reviewers can be "self-rated". However, what happens when they intentionally post high scores all of the time? They are effectively invalidating the system by abusing it.

Originally posted by Richie:

c) I think a review should stand regardless but with no score. Flagged as score removed by the designer. With an average reviewer score and the review itself, readers should be able to detect trolling, but a moderator would certainly help.
Now there's alot of commitment! OK, looks like one vote for the Troll flag.

Admiral Keth

10-22-2004, 12:08 PM

Originally posted by jwxspoon:
Admiral - what would be neat wouuld be to have a "point system" awarded for scenario reviews. A new registrant at the Scenario Depot might start with 10 points in his 'account'. As he downloads new scenarios, his account balance is reduced. When he posts reviews, his acount balance is credited. Etc.
jw OK, let me paraphrase to see if I understand...

In effect, everyone starts out with a 10 point credit. 1 download = 1 credit decremented. 1 review = 1 credit incremented.

I like this idea, but the real test will be from the authors. Once the player has consumed his/her 10 credits, they could potentially sign up again (although they would have to use a different valid email address). In addition, will this discourage players from downloading scenarios?

Authors must weigh in on this subject before it goes on the Giant SDv3 To Do List.

Richie

10-22-2004, 12:11 PM

Clearly I'm not up with the editing features available to me as a designer...

"Richie slaps his head"

I'll have to go back and read the rules...

Sooo that's JWX...

"Mental note: Never send picture of me to Kingfish"

junk2drive

10-22-2004, 12:17 PM

It makes me crazy when the designer states not for play as axis vs ai and the reviewer says he played as axis vs ai and the scenario sucked.
Is that trolling?

Sergei

10-22-2004, 12:18 PM

Originally posted by junk2drive:
It makes me crazy when the designer states not for play as axis vs ai and the reviewer says he played as axis vs ai and the scenario sucked.
Is that trolling? Illiteracy?

Admiral Keth

10-22-2004, 12:20 PM

WOW. My first double post.

Richie

10-22-2004, 12:21 PM

It makes me crazy when the designer states not for play as axis vs ai and the reviewer says he played as axis vs ai and the scenario sucked.

No, just a Slaphead at work...

DO'H!

But the designer should be able to therefore discount the score. I remember Rune bringing this up once...

Admiral Keth

10-22-2004, 12:36 PM

Originally posted by Sergei:
How much is 'trolling' a real problem, and how much is it of that people have very different mind sets, experiences etc.? I think there are people who give scenarios weird ratings with odd explanation, but I don't think all of them really are trolls. Just... different.
There have been several blatant trolls in the past, but they wer handled quickly and quietly. This was one of the reasons for implementing the Banning feature now in place at The Scenario Depot.

Originally posted by Sergei:

Maybe you could have the possibility of rating ratings, maybe, in a "was this review useful?" way. If a review has a certain amount of negative ratings, then it would have diminished or no weight in the overall score. Designer couldn't vote (but he can of course leave his response and ask to explain if the review was strange) to avoid him from purposely downgrading the value of negative reviews no matter how reasoned the opinion...I love this idea. However, how will positive ratings affect negative ratings. As an example, does 3 negatives and 5 positives yield a +2 overall Reviewer Rating (RR)? Perhaps as a percentage...1 positive RR = 100% of the score applied to the overall scenario cumulative rating. Then 2 postitive RR's = 100% of the score applied to the overall scenario cumulative rating. A successive negative RR will result in 50% of the score applied to the cumulative rating. Therafter a successive positive will result in 66% of the score applied to the cumulative rating. One negative RR will simply eliminate one positive RR.

The higher the number of positive RR's, the less impact any single negative RR will have on the individual review ratings, thus on the overall scenario cumulative score.

Complex, and I'd need to see what effect it would have on server load, but entirely possible.

Admiral Keth

10-22-2004, 01:27 PM

Another consideration...

Authors may choose to elicit numeric only, text only, numeric and text, or no reviews.

How will text only and no review scenarios weigh against those which are being numerically rated? Should they even count, i.e., if an author wants their scenarios to be ranked against each other on the Top 10 lists, should they be compelled to use numeric ratings?

If so, this will require that there be hard-coded numeric categories (Map Quality), as well as author-defined numeric categories.

If there are hard-coded numeric categories, should they _always_ be mandatory fields on the review page, in additon to anything the author may define?

So far, the design I'm envisioning goes something like this:

* 4 hard-coded ratings for aspects that should be in _any_ scenario. Regardless of what options the author selects (Numeric Only, Text Only, Text & Numeric), these four values will always be displayed and prompted for when a reviewer posts a review. If the author elects to _not_ receive reviews, then the scenario will never be on any Top 10 List.

These ratings will be:
Playable vs Human Opponent
Playable vs. AI
Briefing Quality
Map Design

Again, zero values in any category are not included in calculations and do not count against the average score. These are the values that are used in the Top 10 Lists.

The author can define 0 to 6 additional rating categories which _are not_ included in the cumulative score. These values are for the author's evaluation of the scenario, and are never used to rate one scenario against another.

In addition, scenarios will only be compared to one another based on historicality. Thus fictional scenarios will be in their own Top 10 List of Playable vs. Human.

Frenchy

10-22-2004, 02:14 PM

Admiral...

Too bad a 1 for 1 (review for download) system couldn't have been instituted.

jw Before a person can download a scenario have him submit a review. I am no coding guru but there must be a way.

Richie

10-22-2004, 02:18 PM

In my opinion, for what it's worth...

You want to make it as easy as possible for the reviewer or the reviewie.

The author can choose to include the numerical value for the review or not. Written text stays.
Readers will read all regardless, most of the time. If a designer has discluded several scores it will show as they are flagged.

Include all in the top ten lists after modification, or disclude if they have discluded X number of scores. You need 3 to be in the top 10?

The reviewer must earn the right to review, as JWX suggested. There is an accumulative average score for any reviewer, against an overall total of reviews, so readers know where they stand with any given review and the reviewer will gain credit by being evenhanded as the number of reviews grows.

I would steer clear of further disemination of scoring than you already have. The harder you make it the less people will review.

Can I ask if you get more reviews since v2 or less reviews?

[ October 22, 2004, 11:26 AM: Message edited by: Richie ]

Admiral Keth

10-22-2004, 02:21 PM

Originally posted by Frenchy:
Admiral...

<blockquote>quote:<hr /> Too bad a 1 for 1 (review for download) system couldn't have been instituted.

jw Before a person can download a scenario have him submit a review. I am no coding guru but there must be a way. [/QUOTE]Sure, it's entirely possible. When a new user registers, their downloadcredit field is set to zero. When a review gets posted, it get incremented to 1. Thereafter, any download decrements it.

However, I'm more in favor of setting everyone's credit to 10 when they register (as well as everyone who is currently registered). I'll figure out some method by which people who have contributed existing reviews can have additional credits.

Admiral Keth

10-22-2004, 02:39 PM

Originally posted by Richie:
Can I ask if you get more reviews since v2 or less reviews? v1 was in operation for 21 months with a total of 5879 reviews (CMBO/CMBB/CMAK battles/operations/maps) = 279.95/month

v2 has been in operation for 4 months with a total of 470 reviews = 117.5/month

GreenAsJade

10-22-2004, 07:05 PM

Adm - thanks so much for listening to our feedback and being prepared to contemplate change yet again!

There will be as many different system suggestions as people responding, I'm sure.

The issue of the person being reviewedm not liking reviews is not unique. Even EBay has it, and copes, with an (mostly) automated system.

Here is my suggestion...

1) Get rid of all existing numeric scales and replace them with one: recommendation for the scenario - a rating of 1 to 5.

This is really the most useful thing that you can record a number for - to help users quickly find highly recommended scenarios. This is the thing that has gone away in V2.

2) Have a check box to indicate whether the recommendation is for PBEM or AI, and make both the PBEM and AI ratings of scenarios available.

People are looking for either PBEM or AI, not both at the same time (usually).

3) Have a text review section where people are encouraged to discuss map design, play balance, briefing etc, but not give ratings to those.

SO:

When I review a scenario, I log in, click "PBEM" or "AI", select a number from 1 to 5 as recommendation rating, and optionally provide a text review.

The rating is _clearly_ subjective (so we don't have arguments about the details of how to come up with it) and the review process is made very easy for those who want to provide feedback quickly.

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.

One rating from each designer for each reviewer - a "fairness" number, and some text to explain why. The designers can probably change their rating if things change. Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

Hence repeat offenders become less and less significant in ratings, while becoming more and more obvious in their own rating page.

This moves the argument away from one bad review ... one bad review: put up with it... if they are really a troll, it will emerge.

Whadaya reckon?

GaJ.

[ October 22, 2004, 04:23 PM: Message edited by: GreenAsJade ]

GreenAsJade

10-22-2004, 07:10 PM

Heck, make it so you have to register to download too, so we can see who's downloading heaps and reviewing little...

GaJ.

Pzman

10-22-2004, 07:28 PM

IMO the current system is good. That said, I spend most of my CM time designing/testing so I don't play scenario's made by others all that often, just don't have to time really. I think we could have the current system based on most reviews along with another list with the highest scoring reviews. Then it is up to the player, on how they want to choose. As I said to start though, the current system seems fine to me.

GJK

10-22-2004, 07:39 PM

Originally posted by Admiral Keth:
<blockquote>quote:<hr />Originally posted by Frenchy:
Admiral...

<blockquote>quote:<hr /> Too bad a 1 for 1 (review for download) system couldn't have been instituted.

jw Before a person can download a scenario have him submit a review. I am no coding guru but there must be a way. [/QUOTE]Sure, it's entirely possible. When a new user registers, their downloadcredit field is set to zero. When a review gets posted, it get incremented to 1. Thereafter, any download decrements it.

However, I'm more in favor of setting everyone's credit to 10 when they register (as well as everyone who is currently registered). I'll figure out some method by which people who have contributed existing reviews can have additional credits. [/QUOTE]You've got to give credit when a member joins or else they wont be able to download a scenario in order to review it.

Perhaps a "Reviewers Guideline" is in order as well. I personally tend to just go middle of the road unless something exceptional stood out in a positive or negative way, but I can guarantee that if I stacked up a couple of my reviews together, I would see several instances of "hmmm, gave that map a 5 and this one a 4, yet the second one was actually better in my opinion".

GreenAsJade

10-22-2004, 08:36 PM

Originally posted by Pzman:
IMO the current system is good. That said, I spend most of my CM time designing/testing so I don't play scenario's made by others all that often, just don't have to time really. I think we could have the current system based on most reviews along with another list with the highest scoring reviews. Then it is up to the player, on how they want to choose. As I said to start though, the current system seems fine to me. I don't doubt that the current system is fine for people who just want to put scenarios _in_.

It's just not as good as it used to be for people who want to quickly find good ones to take _out_.

Emar

10-22-2004, 08:42 PM

Gotta say I like the depot the way it is now. I think the reviews now are a lot more honest now that there is no motivation to either promote or demote scenarios to make the list. Would also think that a particualary good or bad scenario would tend to make the top 20 most reviewed list that is currently featured.

Also I would have to think that now that you are required to log in and give out your e-mail address to post a review that trolling has pretty much dwindled off (have not seen much of Tommy Tiddlywinks or whatever his name was). Before it was entirely possible for anyone to make up a name and post a review and I think that this could be one of the reasons for less reviews now.

Not sure that all folks would want to have to post a review in order to download another scenario. Sometimes I download a bunch but then only play the ones that appeal to me. Would think that this might lead to people posting bogus reviews just so they could download again.

And as for the "best scenario" lists well one mans treasure is another ones trash. And most players only play a scenario once, even though anybody who thouroughly playtests a battle or op knows a CM scenario can play out a hundred different ways. At the CM level of combat balance of a scenario often depends on the success of one or two tanks or priority units. Score a lucky hit on a critical enemy unit and suddenly people are griping about how unbalanced your scenario is. I know from playtesting ops that if you play it 50 times you are likely to see a full spread of results, just as if you were rolling dice.

Having said all that if you do decide to intitute a top 10 list perhaps the worst review or 2 could be ignored in the scoring (but not deleted). Another possibility is that when a review is posted it could be set up to not officially affect the scenario score until the author can reply to it. Then have it set up so the reviewer would have to reply again to confirm or change his review depending on their satisfaction of the authors explanation and rebuttal.

Admiral I certainly appreciate and applaud your efforts for the CM community. There is no other site like yours

Sergei

10-23-2004, 12:47 AM

Originally posted by GreenAsJade:
I don't doubt that the current system is fine for people who just want to put scenarios _in_.

It's just not as good as it used to be for people who want to quickly find good ones to take _out_. I don't understand you.

1) I download scenarios, and it's not harder than before; you always have to read the reviews to make sure it's not just crap praised high by someone with very different taste before you download it

2) If ratings are not comparable, and there's the chance that you'll miss a gem just because one low rating lowers the total score, then why do you want to compare the ratings?

3) You really should be playing more scenarios that haven't been reviewed!

GreenAsJade

10-23-2004, 01:13 AM

1) Yes, you have to read the reviews, but how many do you want to look at before you find ones saying good things? How did you choose which ones to look at?

Before, if you wanted a highly recommended scenario, you could find a set easily, then read a few reviews to finalise your choice.

2) The ratings _are_ comparable. They are just not trying to be scientifically precise.

There are some people who can't handle imprecision. Do we have to have no useful ratings at all because of those few people's limitation?

3) Sometimes I will feel like playing a scenario blind. Sometimes I will want to quickly find a food one.

With the new arrangements I can still find scenarios blind... but I can no longer quickly find a highly recommended one.

Is the problem with the word "difficult"? It's not more "difficult" in that you still just press your browswer button to download. It is, however, far more time consuming to find highly recommended scenarios.

I really miss that - it was what made it good before.

Anyone else agree, or do y'all wonder what I'm talking about, like Sergei?

GaJ.

Mad Russian

10-23-2004, 01:30 AM

Would it be possible to see the entire list of scenarios for each game ranked instead of just the top 20? Or maybe the top 100, since there are 757 CMBB scenarios at this very moment.

What about a feature that shows the scenarios that have gotten the most downloads, for a set period, such as 30 or 60 days? That would be another indicator players could use to choose scenarios.

One line of reasoning, that often comes out in these threads, that puzzles me, are the comments about why cater to the scenario designers. Unless I am mistaken, and I could be, that is why everybody goes to the Sceanrio Depot. To download the scenarios they have uploaded. If you don't make them happy there won't be anything to get all upset over. Now will there? :confused:

Just a puzzle that continues to pop up now and again.

Good Hunting.

Pzman

10-23-2004, 03:32 AM

Yes it is for downloading, but its not a popularity contest either. I like the idea about the top 100 or even top 50 or something along those lines.

As for number ratings why not just have each players rating standing on its own, like now but with numbers rather than a graphic because it's same thing really. Some may like the graphic others the number, why not have both?

Richie

10-23-2004, 10:01 AM

Originally posted by GreenAsJade:
Adm - thanks so much for listening to our feedback and being prepared to contemplate change yet again!

There will be as many different system suggestions as people responding, I'm sure.

The issue of the person being reviewedm not liking reviews is not unique. Even EBay has it, and copes, with an (mostly) automated system.

Here is my suggestion...

1) Get rid of all existing numeric scales and replace them with one: recommendation for the scenario - a rating of 1 to 5.

This is really the most useful thing that you can record a number for - to help users quickly find highly recommended scenarios. This is the thing that has gone away in V2.

2) Have a check box to indicate whether the recommendation is for PBEM or AI, and make both the PBEM and AI ratings of scenarios available.

People are looking for either PBEM or AI, not both at the same time (usually).

3) Have a text review section where people are encouraged to discuss map design, play balance, briefing etc, but not give ratings to those.

SO:

When I review a scenario, I log in, click "PBEM" or "AI", select a number from 1 to 5 as recommendation rating, and optionally provide a text review.

The rating is _clearly_ subjective (so we don't have arguments about the details of how to come up with it) and the review process is made very easy for those who want to provide feedback quickly.

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.

One rating from each designer for each reviewer - a "fairness" number, and some text to explain why. The designers can probably change their rating if things change. Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

Hence repeat offenders become less and less significant in ratings, while becoming more and more obvious in their own rating page.

This moves the argument away from one bad review ... one bad review: put up with it... if they are really a troll, it will emerge.

Whadaya reckon?

GaJ. I reckon you said it all my man!

Admiral Keth

10-23-2004, 11:45 AM

All,

I'm out of town for the rest of today and most of tomorrow. I'll respond in detail to everyone on Monday.

Mad Russian

10-23-2004, 03:41 PM

Originally posted by Pzman:
Yes it is for downloading, but its not a popularity contest either. I like the idea about the top 100 or even top 50 or something along those lines.

As for number ratings why not just have each players rating standing on its own, like now but with numbers rather than a graphic because it's same thing really. Some may like the graphic others the number, why not have both? I don't think it should be a popularity contest either. All too often I see any ills attributed to those whining designers. (Take a look at a couple of these posts for instance...)Whining from either designers, reviewers or players is counterproductive, and not everything that is wrong with the world originated with a CM scenario designer... smile.gif ;) tongue.gif

I think everyone agrees that the system for putting the scenarios on the site is fine. Where the problems start are with the reviews. The designers see them as 90% accurate and honest. That's not bad. With the current system there have been less complaints that I have noticed. Not that I would see them all, but the community isn't bashful about voicing it's opinion, about the number one archive for CM either.

There needs to be a bit more common ground between the designers and the reviewers at the moment I think. The current process is:

1. Post the scenario.

2. Someone reviews it.

3. Designer responds.

That works well when the review is a good one and everybody agrees. The problems arise when the designer doesn't exactly agree with what the review states. IMHO,that is the issue as far as most designers are concerned.

By the way, I post scenarios to the SD, and I review scenarios, so I have an idea about both sides of this fence.

I really like the idea of a reveiwer having some responsibility for his review. I have had a few of the infamous reviews that were bad, as reviewers ripped some of my scenarios, that were for instance, intended for 'Axis vs AI' play and he played it as the Allied player vs the AI. The designer doesn't have to totally lose it and can pretty much address the issue in positive terms.

BUT if there was a way for the designer to rate the reviewer, as Richie suggests, then both parties can be part of the process. A simple check sheet with a comment section is pretty much what we have now. What is lacking, is any form of recourse for the designer, once the review is done.

You have to also allow for a reviewer that simply didn't like the scenario. That isn't a sin. He CAN not like it. I've had some reviews that I attributed to them not liking, not only a single scenario, but quite possibly my designing technique overall. Guess what? I'm never going to make that reviewer happy, because he/she really doesn't like my work to begin with. That's okay. My work isn't going to rock everybody's world. But he/she has a right to that opinion about my work. They played it and took the time to do a review on it.

I have more than the average amount of scenarios posted by other designers on the site. So I see all kinds of reviews. I think in the long run a bad review is balanced out in the wash. If a scenario gets 7 reviews and 1 is bad it won't hurt it that much. What I think is worse is the designers response to a bad review. If all you do is rip the reviewer for daring to state that there may be something less than appealing with your creation you have done the entire CM communtiy but especially yourself a disservice.

I put at least 40 hours a week into CM and that includes what time I spend at the Scenario Depot.
That is probably above average too as most people's job situation won't allow that. (There are times I wish mine didn't.)I see every review put on the SD and at TPG. I read them all and all the responses. I want to learn what I can about the hobby.

I have read EVERY REVIEW ever posted on the Scenario Depot, to see what I can learn, where other designers made mistakes, to try to avoid making those same mistakes myself. Of course, I was in search of scenarios to play, as well. After awhile, you find certain reviewers that you agree with, and those that you don't.

I applaud Keith for wanting to improve The Scenario Depot and for all of you for airing your concerns and ideas. This is constructive and a very healthy way to make improvements. Just griping to the ends of the earth is counterproductive.

If there were more ways to view the data that reviews generate like;

1)a top 50 or 100 download list,

2)a list that shows the most downloaded scenarios for a set period for say 30-60-90 days,

3)a reviewer list so I could go and see if my favorite reviewer has played something he likes, and let the designers have a say at some sort of feedback for the reviewer.

4)there could also be a list of designers that show the number of scenarios they have done by game CMBO-CMBB-CMAK and how many reviews that they have. It could give you an idea of whether that designer is popular or not. If they will reveiw your work, they either really like it, or really don't like it. Reading the reviews will tell you quickly which it is.

There could be a number of things done, to improve the lot of designers as well, but I really think that they are unneccessary. The designers bring their sceanrios to The Scenario Depot in droves now, there are over 700 CMBB battles alone... :D :D

Good Hunting.

[ October 23, 2004, 12:59 PM: Message edited by: Panther Commander ]

General Bolt

10-24-2004, 01:19 AM

The main thing I like in a game is balance and a good fight. Walk overs suck and being walked on REALLY sucks.

I like to see a review where the user reports type of game (AI or 2 human) and percentage of victory for each side.
Allied H-64% vs Axis-AI 36%
Allied-H 29% vs Axis-H 81%.

There could be a list of all reviews and a total/averaging of the reports.

5 AI battles - Allied 67% Axis 33%
3 two man battles - Allied45% Axis avg 55%

If there was a good number (5 or more) of reports on each scenario then a downloader could determine which scenarios were balanced and which were not.
The newbie could pick a battle he'd be sure to win. The vet/regular could pick something balanced. And the elite CM player could play the same battle as the newbie but pick the difficult side.
Additionally, the author could review these stats and revise his scenario to make it more balanced.

danielh

10-24-2004, 11:51 AM

Admiral Keith,

First, thx for the great job of keeping Scenario Depot at live !

You're absolutely right with the Review scoring !
My Scenario (Stalingrad The Moviescene) which got quite good reviews, received two 0 point reviews by an ***hole who quoted having made the same, although mine was already there for a year and a half !
Now there are maybe 8-10 CMBB Scenarios coming in a week and it really doesn't take much time to download 100kb. Most of the time a look at the map already tells a lot about the game, and whether you may like it or not.

Greets
Daniel

Pzman

10-24-2004, 01:03 PM

Originally posted by General Colt:
The main thing I like in a game is balance and a good fight. Walk overs suck and being walked on REALLY sucks.

I like to see a review where the user reports type of game (AI or 2 human) and percentage of victory for each side.
Allied H-64% vs Axis-AI 36%
Allied-H 29% vs Axis-H 81%.

There could be a list of all reviews and a total/averaging of the reports.

5 AI battles - Allied 67% Axis 33%
3 two man battles - Allied45% Axis avg 55%

If there was a good number (5 or more) of reports on each scenario then a downloader could determine which scenarios were balanced and which were not.
The newbie could pick a battle he'd be sure to win. The vet/regular could pick something balanced. And the elite CM player could play the same battle as the newbie but pick the difficult side.
Additionally, the author could review these stats and revise his scenario to make it more balanced. I cannot see that system working since not everyone plays the game at the same level so scores like that are based on the players ablities, not the quality of the battle itself.

GJK

10-24-2004, 01:07 PM

Originally posted by General Colt:
The main thing I like in a game is balance and a good fight. Walk overs suck and being walked on REALLY sucks.
....
Additionally, the author could review these stats and revise his scenario to make it more balanced. This is all a matter of having a scenario playtested before it is sent to TSD, IMHO. A perfect world would have all scenarios sent to TSD having been playtested for balance so that the reviews left at TSD would be directly related to the fun factor of the scenario only.

Fredrock1957

10-25-2004, 11:43 AM

First off - another big hats off to Admiral Keth for the great site he has given to the CM community.

Being both a designer and a reviewer I believe the site as it is now is functioning just fine (I also believed that the previous version was fine also)...

When I go to TSD I am looking for the following things as a Designer ...
 Ability to upload a scenario with ease. Be able to provide the community with pertinant information about the scenario. Ability to see reviewers feedback. Ability to provide reviewer with designers feedback. All of these seem to work very well...

From the Designers side if someone doesn't like my work... so be it... its not going to annoy me... I am not in the business to have the 'most scenarios' or the 'most popular scenarios' or any other competitive type rating.. I design scenarios for fun... and the community should consider themselves lucky that the designers are willing to share their work with others...

Now from the point of view as a Reviewer/Player ...
 The ability to download a scenario with some knowledge of what it is about (PBEM or Historical or Large or Small, etc). I want to see what others think about said scenario. The ability to provide the designer with feedback about his scenario. These items all seem to work for me...

As a reviewer/player I feel that TSD provides me with the information I need to make an informative choice for the specific type scenario I am looking for... For me I don't really care what others have said about a scenario (in fact I dont like to look at that)... I can read the briefings, look at the map and make my own determination if its good or bad... in fact, in playing some scenarios that have not been rated I have found some really hidden jewels... I also always try to provide feedback on scenarios that I have chosen to play...

So what are the issues...
Best I can tell is the designers want their scenarios reviewed more... That would be nice... but if you restrict downloading to those who are only going to submit reviews then I think you are hurting the idea of TSD...

The reviewers want easier ways to pick scenarios... IMHO it doesnt get much eaiser then it is now... Lets face it in a perfect world.. everytime a scenario would be downloaded, it would get a perfect review, cause it is a perfect scenario, and the everybody would be happy... NOT...

Ok, little things can be refined or tweaked to enhance almost anything... but IMHO TSD works just fine...

Now that I rambled (after having a 10 cup'er of coffee this AM) I will get back to my original thought... THANK YOU KEITH MILLER FOR THE AWESOME SITE it is a pleasure to have people like you, GJK from The Proving Grounds (http://www.the-proving-grounds.com), Todd Johnston from Combat Mission Modification Database (http://www.cmmods.com/) and all the others who donate their time and effort to make this such a great community...

That is all...

-FR

flammenwerfer

10-25-2004, 11:46 AM

What about using the old system with a challenge provision, allowing scenario authors to 'challenge' any review that is trollish or did not follow the battle guidlines, for example.

Also, reviewers would give an overall score from 1-10 which would be culmative.

I think we should keep things as simple as possible, for the reviewer.

junk2drive

10-25-2004, 12:08 PM

Unfortunately so few people post reviews we may be fretting for naught.
I use the author's description more than the reviews. Sometimes if there is a bad review I will try the scen to see if the review is bogus.

Bannon DC

10-25-2004, 10:49 PM

Good discussion.

A few points I would like to add or reiterate.

* True -- few people do submit reviews. I like the idea of encouraging people to submit reviews, but making it manditory on a 1-1 basis will be a total nuisance and most users of the site will howl with complaints.

* To sort out the crack pots who submit lame or bogus reviews, you may institite some sort of "rate the review" system much like you would see on Amazon.com. A simple 1-5 scale works. A user who accumulates enough "low" votes that the reviews he submits are worthless would suffer some sort of sanctioning. Maybe all his future rating are "null" and simply are not counted in the total. Maybe his future posts carry some sort of red flag or a clown face icon. Maybe something else.

* An important point for ANY rating system for reviews or scenarios -- don't disclose ALL the rules or factors that go in to the ratings. This way, you make harder for crack pots to "game" the rating system. Using the example above, don't disclose that it might take 20 (or whatever number) "low" votes before a user's ratings are null. This way, this crack pot can go right on being a crack pot but not know that his lowball rankings are not having an effect.

* A "top #" list is a good idea. Maybe a "top downloads" or "top rated". Is there a way to not count downloads more than once from the same user? (Once again, if some crack pot wants to keep hitting the download button in an effort to jack his numbers up, don't disclose that the system only counts one per user. Let him waste his time.)

* Whatever you do, keep it simple! Getting a couple of moderators to help is a great idea as well.

GreenAsJade

10-27-2004, 08:10 AM

Anyone other than me think it's harder now to find good scenarios to play? We've heard from people who think it's still fine. What about any others?

And what about the detailed suggestion I made for change: nice & simple, and addresses the problems. Any comments on that? What's wrong with what I suggested?

GaJ.

DEY

10-27-2004, 09:22 AM

I think the review system that is in place is fine, if people want to post a review then they do so, if not, thats fine also. Everybody has a different idea of what or how a battle should play out like. If you only download a battle because of a review then I would call that stupid. If I'm made to review a battle before I can download other battles then I won't be around that site anymore.

As far as I'm concerned, what needs a major overhaul at SD is in the author administration area, this as it is, is a nightmare. As it is right now I'm not even going to attempt to upload anymore battles, or update the ones I have there, for that matter either. I only hope that the things keith has said would be done in that area, well be done, because that to me needs immediate attention.
Those things are the ability to overwrite previous battles, able to delete previous battles. able to change data fields in the scenario data page without having to upload the battle again.

I hope your job prospects have worked out well for you Keith.

Admiral Keth

10-27-2004, 10:07 AM

Originally posted by DEY:
I think the review system that is in place is fine, if people want to post a review then they do so, if not, thats fine also. Everybody has a different idea of what or how a battle should play out like. If you only download a battle because of a review then I would call that stupid. If I'm made to review a battle before I can download other battles then I won't be around that site anymore.Point taken.

Originally posted by DEY:

As far as I'm concerned, what needs a major overhaul at SD is in the author administration area, this as it is, is a nightmare. As it is right now I'm not even going to attempt to upload anymore battles, or update the ones I have there, for that matter either. I only hope that the things keith has said would be done in that area, well be done, because that to me needs immediate attention.Yep, it's a little awkward right now; that's wht it's being updated.

Originally posted by DEY:

Those things are the ability to overwrite previous battles, able to delete previous battles. able to change data fields in the scenario data page without having to upload the battle again.
These items are being coded as we speak...er, write.

Originally posted by DEY:

I hope your job prospects have worked out well for you Keith. So far, no dice. Although they have extended my employment to the end of next January, it's only a Stay of Execution, not a Reprieve.

Admiral Keth

10-27-2004, 10:31 AM

Originally posted by GreenAsJade:
Adm - thanks so much for listening to our feedback and being prepared to contemplate change yet again!
No worries, mate! Every forum member who cares about supporting the author's ability to refine and enhance their work should be participating in this discussion.

Originally posted by GreenAsJade:

There will be as many different system suggestions as people responding, I'm sure.
True, but the simpler, the better. I'm looking for systems which have a broad degree of usefullness to both players and authors, yet are easy to implement and use.

Originally posted by GreenAsJade:

The issue of the person being reviewedm not liking reviews is not unique. Even EBay has it, and copes, with an (mostly) automated system.

Here is my suggestion...

1) Get rid of all existing numeric scales and replace them with one: recommendation for the scenario - a rating of 1 to 5.

This is really the most useful thing that you can record a number for - to help users quickly find highly recommended scenarios. This is the thing that has gone away in V2.
Something similar to (I know it's atrocious English, but...):

My Scenario Recommendation

1 Excellent
2 Good
3 Fair
4 Below Average
5 Poor

Everyone MUST keep in mind that ANY numeric scale is subjective to the reader. Thus, it will be incumbent upon the reviewer to clarify their position through the text review portion.

Originally posted by GreenAsJade:

2) Have a check box to indicate whether the recommendation is for PBEM or AI, and make both the PBEM and AI ratings of scenarios available.

People are looking for either PBEM or AI, not both at the same time (usually).
Simple enough.

Originally posted by GreenAsJade:

3) Have a text review section where people are encouraged to discuss map design, play balance, briefing etc, but not give ratings to those.
OK, although eliciting that information from people is usually like pulling teeth. This can be alleviated by me providing more explicit instructions above the text area, which ddescribes the type of information the player should be discussing. Conversely, I could create field-level help which provides information specific to the field in question.

Originally posted by GreenAsJade:

SO:

When I review a scenario, I log in, click "PBEM" or "AI", select a number from 1 to 5 as recommendation rating, and optionally provide a text review.

The rating is _clearly_ subjective (so we don't have arguments about the details of how to come up with it) and the review process is made very easy for those who want to provide feedback quickly.

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.
One rating from each designer for each reviewer - a "fairness" number, and some text to explain why. The designers can probably change their rating if things change. Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".
I'm liking this, although I also forsee implementation of the "Was This Review Helpful?" for the players. Thus, both players and authors get to rate the review.

Originally posted by GreenAsJade:

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.
It's early and I'm having some trouble following this. I'll have to re-read it more carefully later on. I'm also a little concerned about page load time with the number of calculations that would need to be performed.

Originally posted by GreenAsJade:

Hence repeat offenders become less and less significant in ratings, while becoming more and more obvious in their own rating page.

This moves the argument away from one bad review ... one bad review: put up with it... if they are really a troll, it will emerge.

Whadaya reckon?

GaJ. Many of these thoughts do have significant merit. Let's see how they either stand on their own, or can be meshed with other ideas.

Admiral Keth

10-27-2004, 10:41 AM

Originally posted by GreenAsJade:
Heck, make it so you have to register to download too, so we can see who's downloading heaps and reviewing little...

GaJ. Is this something authors really want? Although it is easy to implement, what would the authors want to do with the data? Sure, the authors can get a list of people who downloaded your scenario, but do the authors want to then begin pestering players for reviews?

Even if there is simply a list of usernames and download counts ratioed to review counts, that's a feature that might cause many people to stop downloading and playing altogether.

This will really have to be justified by the authors and approved by a large number of players prior to implementation.

Admiral Keth

10-27-2004, 10:46 AM

Originally posted by GJK:
You've got to give credit when a member joins or else they wont be able to download a scenario in order to review it.
Agreed. So far the credit count is 10 free downloads prior to needing to input 1 review. Anyone want a different ratio? Perhaps 1 review = 2 DL credits.

Originally posted by GJK:

Perhaps a "Reviewers Guideline" is in order as well. I personally tend to just go middle of the road unless something exceptional stood out in a positive or negative way, but I can guarantee that if I stacked up a couple of my reviews together, I would see several instances of "hmmm, gave that map a 5 and this one a 4, yet the second one was actually better in my opinion". Field level help would go far to assisting with this idea. This would require criteria from authors as to the nature of the information they want they reviewer to enter.

Admiral Keth

10-27-2004, 11:02 AM

Originally posted by Emar:
Gotta say I like the depot the way it is now. I think the reviews now are a lot more honest now that there is no motivation to either promote or demote scenarios to make the list. Would also think that a particualary good or bad scenario would tend to make the top 20 most reviewed list that is currently featured.
The implicit desire for some authors to have their scenarios _always_ at the top of the lists was one of the stronger reasons why the Cumulative Rating System was abolished. There _were_ authors who wanted especially low rating reviews excised in order to increase their standing.

Originally posted by Emar:

Also I would have to think that now that you are required to log in and give out your e-mail address to post a review that trolling has pretty much dwindled off (have not seen much of Tommy Tiddlywinks or whatever his name was). Before it was entirely possible for anyone to make up a name and post a review and I think that this could be one of the reasons for less reviews now.
Trolling has pretty much been nipped in the bud with having to register, as well as my ability to ban people completely from the site. I have not had to do this yet, but the feature is in place, waitng to utilized.

Originally posted by Emar:

Not sure that all folks would want to have to post a review in order to download another scenario. Sometimes I download a bunch but then only play the ones that appeal to me. Would think that this might lead to people posting bogus reviews just so they could download again.
That's why I'm thinking a different ratio; 1 review = 2 DL's, or 1 review = 5 DL's. I would want to strike a fair balance between requiring reviews and allowing downloads, especially if the player wants to just get a couple for entertainment purposes.

You are correct with regards to the 'bogus review' issue, but that is a possibility now.

Originally posted by Emar:

And as for the "best scenario" lists well one mans treasure is another ones trash. And most players only play a scenario once, even though anybody who thouroughly playtests a battle or op knows a CM scenario can play out a hundred different ways. At the CM level of combat balance of a scenario often depends on the success of one or two tanks or priority units. Score a lucky hit on a critical enemy unit and suddenly people are griping about how unbalanced your scenario is. I know from playtesting ops that if you play it 50 times you are likely to see a full spread of results, just as if you were rolling dice.
True, a player's opinion of a scenario may be biased based on his experience level. A scenario which the author intentionally made challenging for experienced players may overwhelm a novice, thus coloring their response to how the sceanrio was actually to be played. Therefore, they may return an unfavorable review.

To counter this, what about attaching some kind of experience level to scenarios? For example, the author may indicate that this scenario should be played by beginners to CM, or by experienced grogs.

Originally posted by Emar:

Having said all that if you do decide to intitute a top 10 list perhaps the worst review or 2 could be ignored in the scoring (but not deleted). Another possibility is that when a review is posted it could be set up to not officially affect the scenario score until the author can reply to it. Then have it set up so the reviewer would have to reply again to confirm or change his review depending on their satisfaction of the authors explanation and rebuttal.
Hmm. A bit too much rigmarole for people to deal with. Ease of use for both the author and the reviewer is the watchword here.

Originally posted by Emar:

Admiral I certainly appreciate and applaud your efforts for the CM community. There is no other site like yours Emar, thanks. It's a challenge to design a system which is at once useful, easy, and informative.

junk2drive

10-27-2004, 11:08 AM

Not wanting to complicate this further, but I would like a user guideline too. I stink at some tactics. I have started to play some very well done battles, and found them to be way over my abilities. A newbie, average, expert rating would be nice.

Admiral Keth

10-27-2004, 11:12 AM

Originally posted by Panther Commander:
Would it be possible to see the entire list of scenarios for each game ranked instead of just the top 20? Or maybe the top 100, since there are 757 CMBB scenarios at this very moment.
The page return time on a list like that would be prohibitive. It all has to do with the load being placed on the server-side PHP engine, and what it can realistically return in a respectable amount of time before the person who clicked on the link gets bored or thinks it locked up.

Originally posted by Panther Commander:

What about a feature that shows the scenarios that have gotten the most downloads, for a set period, such as 30 or 60 days? That would be another indicator players could use to choose scenarios.
Possible, but again, I'd be concerned about page return time. I can code it up and give it a try, but this one will go somewhere near the bottom of The Giant List of Sceanrio Depot v3 Things To Do.

Originally posted by Panther Commander:

One line of reasoning, that often comes out in these threads, that puzzles me, are the comments about why cater to the scenario designers. Unless I am mistaken, and I could be, that is why everybody goes to the Sceanrio Depot. To download the scenarios they have uploaded. If you don't make them happy there won't be anything to get all upset over. Now will there? :confused:

Just a puzzle that continues to pop up now and again.

Good Hunting. There must be a balance between the author's ability to glean useful information from reviews, and players ability to place those reviews, offset by some burden upon the player to come back and contribute. So far reviews have been 100% voluntary, but there seems to be some call for semi-compulsory reviews. The jury is still out.

vulture

10-27-2004, 12:30 PM

I have to agree with GreenAsJade to some extent. I've only used the Scenario Depot a few times (being a relatively recent and fairly infrequent CM player) but I found the previous system much more convenient in terms of finding a good scenario of the type I want to play. Today, for example, I want to look for an Allied Attack scenario (Medium size), which can be pretty well vs an Axis AI defender. That gives me 62 options, which I then have to look at one by one to find which can be played well vs Axis AI defender. With v1 of the Scenario Depot, it was much easier to find such a scenario quickly. It could be ameliorated to some extent by allowing an extra search field for 'playable vs Axis AI', and it being up to the designer to give fill in some kind of field for whether a scenario is suitable for that kind of play.

My other problem is that I haven't been able to use the scenario depot at all from the computer I actually have Combat Mission installed on. It's Windows XP, pretty much straight out of the box, but clickingon those nice buttons such as 'download', 'submit review', or anything else with that style of button completely fails to do anything. Now although this is almost certainly the fault of my security settings in some way, it's also the only site on which I've noticed any problems at all.

Mad Russian

10-27-2004, 11:44 PM

Originally posted by vulture:

My other problem is that I haven't been able to use the scenario depot at all from the computer I actually have Combat Mission installed on. It's Windows XP, pretty much straight out of the box, but clickingon those nice buttons such as 'download', 'submit review', or anything else with that style of button completely fails to do anything. Now although this is almost certainly the fault of my security settings in some way, it's also the only site on which I've noticed any problems at all. I use a computer that has XP straight out of the box too. I have CMBB and CMAK loaded on it and have no problems accessing the Scenario Depot. I don't think XP is your problem.

Good Hunting.

Bannon DC

10-28-2004, 12:33 AM

Originally posted by Admiral Keth:
<blockquote>quote:<hr />Originally posted by GreenAsJade:
Heck, make it so you have to register to download too, so we can see who's downloading heaps and reviewing little...

GaJ. Is this something authors really want? Although it is easy to implement, what would the authors want to do with the data? Sure, the authors can get a list of people who downloaded your scenario, but do the authors want to then begin pestering players for reviews?

Even if there is simply a list of usernames and download counts ratioed to review counts, that's a feature that might cause many people to stop downloading and playing altogether.

This will really have to be justified by the authors and approved by a large number of players prior to implementation. [/QUOTE]This would not be important to me and I don't see what value it would be. This feature is in use at The Proving Grounds and I think it makes sense over there given the scope of the site. The point of that site is to post your scenarios and have them playtested. Even though that is the purpose of the site, it is still hard to get feedback.

A "shame-o-meter" is too much mothering. Plus, you need to keep in mind that many users of the SD are not native English speakers and may have a hard time being forced to submit reviews.

To me the SD is more of a "mass market." That is why I think some sort of rankings of the scenarios is a good idea. There are a lot of good scenarios, a few great ones... and some that should not have been posted because of major flaws. I DO want to know about the great scenarios because I don't have time to DL and look at all of them. I don't want to get 20 turns into a big battle to find out it is unwinnable.

Here's an idea for encouraging reviews -- monitor the amount of reviews people submit and create some sort of level or ranking system for the users. For example, the user posts 20 reviews, and he earns the title of "Lieutenant." (similar to what other sites do). The REWARD for submitting reviews is that you are allowed to vote for the top ranked scenarios... or get to pick the "scenario of the week" as your rank increases. Moderators can monitor the worthiness of reviews and weed out any crack pots who are spamming the site just to gain rank. This way responsible people who have an eye for good scenarios will be the ones who are voting on the top scenarios.

With this system hopefully you'll get more of a variety of scenarios on this list.

You could still do a "top downloads" list which also interests me. (I would be interested in seeing at least the top 50 -- with # of DL's, even if I had to click a few times to see the whole list.)

Pzman

10-28-2004, 01:08 AM

There already is a top ten download list, but that does seem a little short. A rank system sounds somewhat interesting to a point. As for bad scenarios, well that may be so, but last time I checked there were no requirements for the level of design required to post there, nor IMO should there be. As has been said before, one mans garbage is another mans tresure.

To me any rating system, although useful, means that a lot of good scenarios may be missed because nobody takes the time to review them. There are still hundreds of battles without reviews, but that does not mean they are any better or worse than the others.

vulture

10-28-2004, 05:51 AM

Incidentally, another argument against requiring reviews before downloading more scenarios: Consider someone off for a few weeks with their laptop and no internet access guaranteed. They might just want to download a reasonable groups of scenarios (say 5 in one go), with the intention of playing (and reviewing) them all over the next few weeks.

GreenAsJade

10-29-2004, 05:24 AM

Originally posted by DEY:
I think the review system that is in place is fine, if people want to post a review then they do so, if not, thats fine also. Everybody has a different idea of what or how a battle should play out like. If you only download a battle because of a review then I would call that stupid.

I guess you'll have to call me stupid. It worked really well for me though: downloading highly recommended scenarios always got a good result,
and quickly.

It amazes me how much time some people have to go about searching for a good one and reading reviews to make the selection. Good luck to you. I would like to have a quicker way to get to the good ones!

If I'm made to review a battle before I can download other battles then I won't be around that site anymore.

Although I think I've reviewed every battle I've played, I have to agree that forcing people to do reviews won't do anyone any good. It will just generate rubbish reviews. I want to read reviews by people who wanted to write them.

GaJ.

GreenAsJade

10-29-2004, 05:31 AM

Originally posted by junk2drive:
Unfortunately so few people post reviews we may be fretting for naught.
Actually, this is a regular "complaint" ... "we don't get enough reviews", but I would say 100+ a month is not too shabby...

GreenAsJade

10-29-2004, 05:32 AM

Hans

10-29-2004, 06:01 AM

Looking at other organizations that try to judge independent artistic creation. I would think an ad hoc group, selected by the Admiral. Would conduct an academy awards type arrangement. Rating games would come from three aveneues

1. The Grog core, restricted to those designers with say more than ten scenarios and selected by one another (very cliquely) They would recommend scenarios for ratings in various catagories.
2. All registered Forum users (using a voting system to be added-perhaps a top ten?)
3. Anyone who puts in a review. (using the existing system)

GreenAsJade

10-29-2004, 08:05 PM

Originally posted by Admiral Keth:

Something similar to:

My Scenario Recommendation

1 Excellent
2 Good
3 Fair
4 Below Average
5 Poor
Simple and effective: works for me.
(Though I would make a good score higher
than a bad one)

Or

5 Must Play
4 Highly Recommended
3 Worth Playing
2 You might like it, I didn't.
1 Fatally flawed: don't bother.

Originally posted by GreenAsJade:

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.
(snip) Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".
...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

It's early and I'm having some trouble following this. I'll have to re-read it more carefully later on. I'm also a little concerned about page load time with the number of calculations that would need to be performed.
I thought an example would illustrate best, so I emailed one to Admiral. Then I thought "what the heck why not let everyone pick it to bits. So here it is:

Joe the reviewer has posted 3 reviews. The designers of each scenario he reviewed
gave him ratings of 4, 5 and 3 (out of 5) respectively as a reviewer. They thought his reviews were pretty good.

So his personal reviewer rating is (4+5+3)/3 = 4.

Troll the reviewer has posted 2 reviews. The designers of each scenario he
reviewed gave him 0, 1 respectively.
He was obviously a troll.

So his personal reviewer rating is (0 + 1)/2 = 0.5.

Joe and Troll each come to review a new scenario.

Joe gives the scenario a recommendation of 4 ("highly recommended").

Troll gives the scenario a recommendation of 1 ("don't bother playing it").

The net recommendation for the scenario from these
two reviews is the is the average of Joe's recommendation and Troll's recommendation
weighted by their reviewer rating .

Calculation:

[ (Joe rating * Joe Score) + (Troll rating * Troll score) ] /
(Joe rating + Troll rating)

which is

[ (4 * 4) + (1 * 0.5) ] / (4 + 0.5) = 3.7

See how this scenario ends up being nearly highly recommended (3.7 is nearly 4), as
rated by Joe, because Joe's recommendation counted 8 times as much as Troll's. But it's not quite a 4 on the off chance that Troll was pointing out a real flaw.

Now suppose Joe goes psycho, and starts submitting bad reviews. Before long
his own reviewer score average drops to Troll like levels and he can no longer
significantly affect the ratings of scenarios.

Note how all this happens without a moderator having to be involved.

By rating reviewers each time they submit a review, the scenario designers have the power over time to rule out trolls,
without individual reviews having to be assessed and argued about.

Why I'm putting this forward:

1) It gives a system to let people quickly find highly recommended scenarios.

2) It makes reviewers accountable for their reveiws.

3) It lets questionable reviews be dealt with without a moderator having to put effort into sorting through an argument about individual reviews.

GaJ.

Pzman

10-29-2004, 08:36 PM

So if someone posts low scores more than once that makes them a troll? Hmm I don't know, but it must take a while to figure it out. I mean even trolls will like some scenarios and give them 3-5 of 5. One thing that bugged me with the old system for example was when someone would post the same low scoring review twice. Now maybe there was a double post problem like there is here sometimes. If that is the case, rather than being marked as a troll, the person doing the review should have the option of deleting their own review. Or another option would be to only allow one review per person per battle.

junk2drive

10-29-2004, 09:39 PM

I have a theory about generational differences.

Akula2 and I had a discussion about fixed or locked down placement of units in setup. He believes the player should always have the choice of where to place his units in setup. I can see a designer, after playtesting, deciding to lock units for balance of gameplay.

I am in my 50s. I don't know about Akula2 and it doesn't matter. There is nothing wrong with his manner of playing or his preferences.

My theory is that I grew up (sort of) with checkers, chess, and games with defined placement of pieces. Baseball allows some placement of players, but not total freedom. So I am use to being told, so to speak, where to start from. In fact it is more enjoyable sometimes, than fretting over what to do. Younger people raised on console games, may have a different outlook on what wargaming is. No right or wrong here, just different.

How would these two types of players rate the same battle?

GreenAsJade

10-29-2004, 10:14 PM

It doesn't matter how each of them rates it.

It's a shame that designers will feel so passionate about any given individual rating (because it is about their work, afterall) when for the purposes of selecting highly recommended sceanrios its all about averages.

If you give it 4 and Akula2 gives it 2 then it will show up as 3. That will at least position it and allow someone to start making an assessment. Someone else will read _why_ you gave it 4, and will choose to play it because you made sense to them. They will give it 4. It's average will move up to 3.3 (or whatever). Now it is already rated better than average because at least _some_ people liked it, so it is a more logical choice for someone looking for a scenario than one which no-one liked.

It's all about averages. Don't get hung up on individual score.

GaJ.

GreenAsJade

10-29-2004, 10:18 PM

Sigh - the joys of written communication!

Originally posted by Pzman:
So if someone posts low scores more than once that makes them a troll? Of course not.

Notice that I didn't say what Troll gave as scores at all. What I said was that the designers gave _him_ low scores, because they thought his reviews stank.

If someone posts a well thought through and explained review where they give a low score, hopefully most designers will be mature enough to give the reviewer a reasonable rating.

What makes someone a troll is a review with a low score and text that says "This scenario sucks. The author's mother stank of elderberry wine."

Don't confuse the rating that someone gives to a scenario in a review with the rating they get back in turn from the designer.

In my example above it was assumed that the two low reviewer_ ratings that Troll got were because his reviews stank, not just because they were low scores.

GaJ.

[ October 30, 2004, 05:10 AM: Message edited by: GreenAsJade ]

GreenAsJade

10-29-2004, 10:20 PM

This kind of stuff is self correcting, by the way. That's the beauty of it.

If I give your scenario a long well thought out review explaining why it wasn't enjoyable, and you give me a low reviewer rating, I'm not going to review any more of your scenarios, am I? Problem solved: I get good reviewer ratings from people who appeciate my review style and you don't suffer from reviews from me that you hate.

GaJ.

junk2drive

10-29-2004, 11:16 PM

Good points.

GreenAsJade

10-30-2004, 08:07 AM

(ooops, mistake post!)

c3k

10-31-2004, 07:06 AM

Gents,

As a recipient of the scenarios produced and presented on the scenario depot, I am against any kind of mandatory review process.

I've downloaded more scenarios than I've reviewed. However, I've posted a review for EVERY scenario I've played (which I remembered was from the Scenario Depot). Perhaps appending a Scenario Depot reminder to the briefing would help?

Making it mandatory (in whatever ratio) will, according to the law of unintended consequences, NOT result in meaningful reviews being posted.

I agree with the poster who asked for a toggle for "best played as PBEM/AI" with the various roles explicitly delineated. Indeed, I would volunteer to help go through the database and find and sort the various scenarios in such a manner.

Simpler ratings? Yeah, that'd be nice. A tighter search function? Toss it in! Ease of use, drop down menus, all those items that make a visit to the site quick, painless, and rewarding are what will keep a lot of reviews coming in.

"Thanks" to all you talented designers (even the ones just learning) and to Keith for the site. This is what keeps the CM series alive for me.

Regards,
Ken

(edited for clarifications.)

[ October 31, 2004, 09:18 AM: Message edited by: c3k ]

jwxspoon

10-31-2004, 12:34 PM

Ken,

In the proposed scheme, a reviewer would get multiple credits for a review, which he would then spend on multiple scenario downloads.

As a designer, it is quite irritating to have literally thousands of downloads and only 1 or 2 reviews. It helped to hasten my decision to stop designing scenarios for public distribution.

I applaud the fact that you have reviewed every scenario you have played. You are just the kind of player I designed my scenarios for. Unfortunately the large majority simply play and move on. While I did not expect a plethora of reviews for my scenarios that came out on the various CD's, it would have been nice to get more feedback from the extras scenarios I did to help keep the game going.

In any case I think that Admiral Keith's Scenario Depot has proven to be one of the most valuable 3rd party resources ever created for CM. I applaud anyone, whether they be designers or people like Keith that help extend the life of a great series of games.

jw

junk2drive

10-31-2004, 12:54 PM

Some actors become directors. Probably a control freak thing. I play for something to do. I mod for my enjoyment. I share those mods and feed my ego. I don't care if someone downloads my mod and deletes it. I don't care if 1 or 100 people download a mod, but it is nice to see 100. I may move up to creating scenarios someday. I hope I feel the same about my scenarios as I feel about my mods. Reviews are nice, but not a necessity for my stuff. I share to share.

markl

10-31-2004, 02:59 PM

I agree with junk2drive above. Although I must admit I do not contribute just download.

If you see your scenarios downloaded a lot of times that should indicate people like them. I do not need a lot of reviews to know that say "Hans Small Battles" are ok as I have followed them for a long time. I never look at reviews as I think I play for other reasons that a lot of other people.

I make it one of my criteria to download a scenario from someone I have not played before each time I drop by as you never know who's work you wil like without trying it.

I also download a few at a time. If I had to write a review all the time I would go to another site.

I will try to wite a few more reviews if it is needed to keep people designing, but to me a poor review is a waste of hard disk space, and I must admit when I have re read the few reviews I have made they are not too good.

But since I am only a downloader I will try to do what the designers want as I really appreciate their efforts. They make the game for me.

Cheers

Richie

10-31-2004, 07:17 PM

You get further in life with a gun and a smile than you do with just a gun.

It's far better to encourage people to review than it is to force them. If people have to review to download, it will turn nearly all of them away at some stage.

A reward system for reviews might be the best bet. Perhaps a rank and some medals as suggested here earlier, or the chance to vote for best scenarios. Maybe a quick reply form to make it easy.

Reviews for a designer are an essential tool, like a litmus test, or the skewer you stick in a cake to tell you if it's cooked.
In order to know if the scenario has worked it must be played out, with a million variables and a thousand different outcomes.
The more feedback you have the better equiped you'll be at achieving game balance and creating a challenge for the players...

Sergei

11-01-2004, 02:09 AM

Originally posted by markl:
If you see your scenarios downloaded a lot of times that should indicate people like them.If only it were as clear as that... the amount of downloads does not tell why people have downloaded it, nor how they felt after they played the scenario. After all, how are people going to know that the scenario they are downloading is good if there aren't any reviews? Furthermore, if there are no reviews, how does the designer know why people possibly like them?

jwxspoon

11-01-2004, 09:15 AM

I find myself thinking of the old adage "A car given is kept dirty and driven hard, but a car earned is washed and kept in good repair."

It seems to me that a quid pro quo is in order here so that there is value placed on the designers' time and effort. If having to review a scenario every 10 downloads is too onerous a task, then perhaps the player should stick to something really easy, like Quick Battles.

Like some other posters in this thread, I designed my scenarios for enjoyment and as a way of helping out the community and keeping it 'fresh'. Indeed, I only ever released 25% of my scenarios to the public anyway. Players are free to modify my released scenarios, play with the maps, put them in map packs, whatever. The only thing I asked was a review of the original work when you finished, whether or not you liked it.

jw

junk2drive

11-01-2004, 09:57 AM

How about an automatic email system

Dear player,
You have downloaded 10 scenarios without any reviews. What kind of ********* are you?

Or something more civil

jwxspoon

11-01-2004, 10:18 AM

LOL!!!

c3k

11-01-2004, 11:10 AM

jwxspoon,

Initially I agreed with the proposal to force a review in order to download 'x' scenarios. After some thought, I changed my mind. (As I stated, I do reviews because I appreciate the hard work that goes into scenarios AND because I use the reviews to pick scenarios that I play.)

Now, let's say you need 1 review per 10 additional downloads. The User gets 10 freebies. Great. Now he's hooked. He comes back to the site and gets a message that he needs to post a review for one of the 10 he already downloaded before he can get more. Let's see what happens:

1) He NEVER played any of the 10, they're sitting on his hard-drive. He's come back for 10 more because his pub-mate wants him to play a specific scenario available at the Scenario Depot.

2) He's only interested in downloading every scenario ever created. Some sort of collector/anthology hermit type....

3) He's played each of the first 10 about three times each, but never thought about them from a critical analysis viewpoint...

RESULT: He will post a crap-review just to get more scenarios. The crap-review will help noone. It will fulfill the automated review requirement necessary for him to get more scenarios.

COUNTER-RESULT: In the ever-changing ballet of attack and defense, all reviews are now, themselves, reviewed. This review process requires time. Time=money. The Scenario Depot now becomes a paid subscription site.

COUNTER-COUNTER-RESULT: A gang of Chinese crackers, assisted by out of work former Soviet nuclear weapons engineers, cracks the Scenario Depot encryption and sells the entire collection of scenarios on Ebay.

Or, we just rely on the the community (MOST of whom are kindrid spirits) to post reviews as and how they can.

Regards,
Ken

junk2drive

11-01-2004, 11:28 AM

c3k, so I can tell the wife that playing CM is saving the world by keeping the terrorists busy?
I like that.

Hans

11-03-2004, 06:36 AM

Hey I've worked CM scenario design into the 'on-line learning experience' portion of my CV and as an example of my web exposure ....they are online history modules.

Akula2

11-03-2004, 09:06 AM

I'm an infrequent downloader too.
I never play against the AI unless I've agreed to playtest a scenario.

Whenever possible I try to insist that my pbem opponents pick the scenario. I do this for a couple of reasons that stray from our current topic, but the result is that very often I don't even know what I'm playing. I do, however know that it came from the Depot.
I'll start asking my opponents the names of the scenarios. I'd be happy to post short reviews when we're done.
In fact, I wrote a few quick sentences on jwxspoons Ponyri Express last night only because I found him posting in this thread. In truth, I didnt think he cared either way. He had plenty of opportunity to tell me to go over there and do it every time I came into irc drunk as a monkey.

The depot's greatest tool to get people to post reviews is to make it as easy as possible. I was very pleased last night when I saw that I could rate it with bullets instead of words. It took me 2 seconds and gives both designer and downloader a quick visual of strengths and weaknesses of the scenario.

I would say, start simple.

Advertise how easy it is with a plea to "please review". Make it full screen before or after the download. Make it upon entering the Depot. Plea often and effectively. Stress how simple it is to just check a few boxes.

I have one other point to stress.

Ya know what scenarios I remember?
I remember the ones that I complained bitterly to myself while I was playing them. I remember the ones that made me fight hard and smart.
These are the ones that stick with me. I hated them while I was playing them.
I like the evil ones, I guess. (A Cold Winter's Morning played as German and rune's Hedgehog Hell come immediately to mind)

I love them all now. Can't you see I even remember the names? I can't remember my own name half the time.

Do you guys want me to review these gems directly after I play them? ... or do you want me to wait, cool down and get back my objectivity?

Anyhow, thats my perspective for what its worth.
Maybe it's unique, but probably not.

I also try to get suggestions as to what scenarios to play. This sort of behavior, if exhibited by a number of players will lead to lopsided download numbers for certain scenarios and maybe even designers.
I guess that's the way of the world. Theres no sense in complaining about it. It's probably more effective to do something about it by promoting one's battles by word of mouth at every opportunity.

If somebody like me posts begging for a 2 player scenario suggestion, you should be "Johnny-on-the-spot", elbow your way into the thread and say, "Hey try this! I made it! Playtesting went well and I think you'll enjoy it! Please post a review at the Depot when your finished. Reviewing is easy. It takes 2 seconds. All ya have to do is check a few boxes. Of course, I would like to know your thoughts also so don't be shy about using the comments section."

I bet they play it. I bet they review it. I bet if they like it they play your other stuff too ... sooner or later depending on how evil it is.

I've seen Battle suggestion threads scroll off this page without a single reply.

My suggestion is to start there and raise awareness about the importance of reviews.

EDIT-
Perhaps after the first 10 downloads and every 5 after that that weren't reviewed could result in an automated email back to the downloader.
This is just a suggestion if anyone is worried about losing people from banning them from more downloads.

The email might be something in the line of, "It's come to our attention that you have downloaded 10 scenarios ... blah blah blah ... importance of feedback to designers ... blah blah blah."

Be a pain in the arse ... often and effectively.
Guilt is your friend!

[ November 03, 2004, 06:31 AM: Message edited by: Akula2 ]

General Bolt

11-13-2004, 01:14 AM

One additional item in the search criteria would be nice. "Best AS" 2-player or AI. Some times I'm looking for a an AI game other times I'm looking for a 2 player battle.

GreenAsJade

11-15-2004, 06:21 AM

Yeah, I've been wanting that for ages!

markl

11-15-2004, 02:36 PM

I also think that not reviews will be useful. What if say I download a historical scenario but find it unbalanced, and the map too flat for my likings. I could post a review stating as much. But that would be unfair as the scenario may have been very accurate in its portrayal of the troops involved and the terrain.

In some circumstances I think the person needs to be qualified to do the review. After all you would not ask just anyone to do a book review if you were going to publish it.

I am not offering this as an excuse for not posting reviews, but as another view.

Cheers MarkL

Carl Puppchen

11-15-2004, 04:46 PM

I have been following this thread and would like to say a few things:

- first, thanks to the admiral for all of the hard work he put into the site! We all appreciate it a lot
- second, thanks to all the designers that spend their time creating scenarios just for others to have fun with. That is a good thing too

with the above said, I think that reviews are nice but they won't always occur. When I first started I was kind of intimidated by the community - obviously you have people jumping in threads like Jason or Michael that seem to know everything - so people don't want to stick their toe out in the water until they really know what they are doing.

There are other forums where the point of the forum is to playtest, such as the proving ground. Part of the beauty of the depot is that they give you scenarios and you can play them or not play them at your leisure. Sometimes I like to download them just to look at the map or the briefing, or to learn something about a particular historical situation.

I think that the people that download these scenarios are generally "good" people who aren't trying to "game" the system. Maybe I am wrong, but this is my opinion.

Given the above items, I would recommend the following:

- put back up the "frequently downloaded" scenarios. It does tell information, even if it is not perfect, and it is objective information. maybe a friend is telling a friend, or they just think that the author writes excellent scenarios, but it is real data.
- put something on the main page site that expressly requests feedback and says that it is important so that authors can do better in the future if people take the time to create reviews. say that it is not required, but it is strongly recommended that people take the time to review scenarios
- perhaps send an email periodically to people on the list asking them to review scenarios that they have played - kind of a friendly reminder - this was suggested above

Just my opinion.

mchlstrt

11-18-2004, 02:35 AM

I happened to come over to this Forum & then I happened to read this whole Thread. I have a take FWIW.

I do Mods. It's good to put something up & have folks say "It's great!", to feel supported & appreciated. However, the people who are going to give me useful feedback are Modders themselves or real Mod Aficionados. People who know what they're looking at, so to speak.

I'm a 'hobby' player. I know I like some Scenarios, but I don't really know why. My input is kinda useless once I get beyond "It's great!'. If I 'had' to review stuff I am exactly the guy who would unintentionally screw up the Rankings before I gave up going there in shame.

What I want to know is what the serious Scenario Designers & Aficionados think.

If someone is into it enough to do X number of Reviews then post their all time favorites, under their name, linked to all Reviews for that Scenario. Vs AI, PBEM, quickies, biggies, whatever Catagories you guys think. But do have a 'Best Kept Secrets' -type one, for stuff that other Folks missed, or that are quirky but kewl. This encourages people to get involved & bring a standard to their input, even a subjective one.

Have 'Hot Picks' lists on the main page based on each 'qualified' Reviewer's latest recommendation in each Catagory, while making it clear that deeper Lists are a click away, under Reviewers names. This keeps the Main Page Lists alive & moving, as it's on a time driven basis. It gives average Schmoes (me) a quick way to grab a Scenario based on something, anything. It also makes it easy to get further (& older) recommendations while drawing people into the whole Review process. When you click on Cap'n Happy's link you know you're getting his opinion, not The Top Ten From On High.

I hope some of this is useful, or at least intelligable.

strt

Erwin Rommel

11-19-2004, 11:56 AM

I for one will never download again from The Scenario Depot if it is implemented that I HAVE to write a review on a scenario to be able to continue to download.Forget that.I wont do anything that I am forced to do.
I think the Depot is just fine as it is.I have no problem sorting through the scenarios to find what I want.
I also would have trouble writing a review as I do not knitpick every little aspect of the scenario..I want to play and enjoy the game not spend the whole time I am playing looking to see if a wheatfield looks like it should or is the proper size for the area being represented.Or whether or not the forces are balanced..I personally dont care about any of that.Im not that pety.I just want to play the game not be a critic.I also dont play PBEM.I play against the AI sometimes,but mostly hotseat and TCIP games with friends locally..and I should not be forced into writing a review just so I can download maps to keep a game I enjoy fresh.

[ November 19, 2004, 09:01 AM: Message edited by: Erwin Rommel ]

Thin Red Line

11-19-2004, 12:41 PM

It's difficult to write a review when you're not a native english speaker.
I suspect this might be the reason why many people don't review.

Philippe

11-19-2004, 04:57 PM

I for one would not want to read most of the reviews that would get written if people had to write them. The quality of what gets written is tenuous enough as it is -- flooding the database with involuntary hack jobs can only make it worse.

It would be nice if every scenario had twenty or thirty reviews -- that would even out the numerical distortions that come from not having a statistically valid sample. But it's not going to happen, and I think it's better if those who care enough to take the trouble to write a review continue to be the ones who do so.

Reviewing is serious business, and shouldn't be foisted on the inexperienced. My own reviews are far too clumsy to be of any use to anyone, and will probably remain that way until I've written thirty or forty of them. Writing clumsy reviews is not my idea of a very efficient or satisfying way to give something back to the community, so I prefer to focus my efforts in other directions.

I think part of the charm of modding is that it is understood to be a voluntary effort -- a freely given gift. Most of the opinions that are given about a particular mod are not of much use to the maker in and of themselves, and even some of the positive feedback is not really very helpful. A modder should only be interested in the feedback of people who know what they're talking about, and should ignore everybody else. Forcing people to write who don't know and who would otherwise have remained silent is probably the opposite of helpful. I can't imagine that the critique and feedback process in the making of scenarios is much different.

What is really needed is not many reviewers, but a few good reviews.

[ November 19, 2004, 02:06 PM: Message edited by: Philippe ]

Kingfish

11-20-2004, 09:26 PM

My apologies if this has been asked already, or if this isn't the proper thread in which to ask, but is it possible to submit a scenario pack as a single entity to the depot? IOW, when they d/l it it will contain all the scenarios of that pack, much like how B & T does with the Stalingrad or ROW packs.

junk2drive

11-20-2004, 09:35 PM

I would think that multiple files in a zip would work.

GreenAsJade

11-20-2004, 11:54 PM

An update: we've heard how some people still love the ole depot, but I've just finished a few scenarios and am finding two things:

1) In the past I've always reviewed scenarios I've played, but I'm finding I can't be bothered writing reviews on the depot because there doesn't feel like much point ... they're just not so useful anymore.

2) I feel more inclined to move to QBs - something I never would have dreamed of in the past - because its now quicker to set up a QB than sift through the depot...

GaJ.

GreenAsJade

11-20-2004, 11:59 PM

I just read at the depot that Adm Keth has V3 redesign underway. Thanks Adm for listening and continuing to improve!

GreenAsJade

11-21-2004, 12:03 AM

While a lot of the discussion has been about the ratings system (and I really hope Adm puts in something to help with the "find a highly recommended scenario easily" goal) there's one other small thing I'd like to ask...

... can we please have more characters in the review boxes! Once someone does decide to do a review, they might as well have room to say what it is they want to say, eh!? I always run out!

GreenAsJade

11-21-2004, 12:22 AM

Oh - another thing to separate out from the "combined scenario score" question: the scenarios definitely need to be searchable by "Best played as".

Each of "the lists" needs to be available as "for AI" and "for PBEM" separately. Otherwise it's forever clicking on popular scenarios, only to find they are popular for the wrong type of play!

[ November 20, 2004, 11:31 PM: Message edited by: GreenAsJade ]

Soddball

11-22-2004, 10:03 AM

Just wanted to add that I haven't got many actually up at the scenario depot (although I have about eight which are finished and which I haven't got around to sending smile.gif ) but I really value the opportunity for feedback on these scenarios and think that the depot is an enormously valuable site.

What about a file included with the scenario that you download. It saves to your computer, and when you've played the scenario you just fill in the info and it sends itself off to the depot?
Might work with some email systems, might not. I don't know, it's just a thought.

GJK

11-22-2004, 11:41 PM

Originally posted by Soddball:
What about a file included with the scenario that you download. It saves to your computer, and when you've played the scenario you just fill in the info and it sends itself off to the depot?
Might work with some email systems, might not. I don't know, it's just a thought. I've chatted with Keith about something similar - a way to export scenario playtest discussions/reviews from TPG over to TSD easily. I had thought of a small, encrypted text file that you could attach to a form post when adding your scenario to TSD but security is an issue here. You're really opening yourself up if you allow outside appends to your database like that (obviously) and having Keith manually do all those appends or merges isn't going to work either. There just doesn't seem to be a way to have it automated -safely.

WWB

11-23-2004, 05:06 PM

Originally posted by GJK:
<blockquote>quote:<hr />Originally posted by Soddball:
What about a file included with the scenario that you download. It saves to your computer, and when you've played the scenario you just fill in the info and it sends itself off to the depot?
Might work with some email systems, might not. I don't know, it's just a thought. I've chatted with Keith about something similar - a way to export scenario playtest discussions/reviews from TPG over to TSD easily. I had thought of a small, encrypted text file that you could attach to a form post when adding your scenario to TSD but security is an issue here. You're really opening yourself up if you allow outside appends to your database like that (obviously) and having Keith manually do all those appends or merges isn't going to work either. There just doesn't seem to be a way to have it automated -safely. [/QUOTE]This really, really screams XML and/or webservice. Should not be too difficult; hardest part will be doing XML parsing at reasonable speed in PHP.

Drop me an email as I have done some similar message passing stuff and have a few clues about how to accomplish this.

WWB

GJK

11-24-2004, 10:23 AM

Originally posted by WWB:
This really, really screams XML and/or webservice. Should not be too difficult; hardest part will be doing XML parsing at reasonable speed in PHP.

Drop me an email as I have done some similar message passing stuff and have a few clues about how to accomplish this.

WWB Yes, XML would probably be the way to go - didn't even think of that! I'll wait until the holidays are over with and see if Keith would be interesting in perhaps working something out together with your help if possible.

Freiheit

11-26-2004, 05:06 PM

My only but really big wish for the Depot's v3 would be the new search category: Best played as.
Good luck, Admiral!

Hans

11-27-2004, 02:29 AM

Hey Keth

I think the curse of bad commo between our two IPs has hit us again. I've sent you some emails but you have not replied. So I'll put a note here in case you come by.

Just a small problem with the Operation submit function and later the Author edit for submitted operations. I'm unable to edit the authors name in Operations.

Thanks

The Hound

11-29-2004, 05:14 PM

Just wanted to say thanks to Admiral Keth for the great resource TSD is. Can it be improved upon? Sure, what can't be.

But let's realize what we have here. You want a particular battle, PBEM or against the AI, large or small, historical or fictional, in whatever theater it's all there.

GREAT JOB!!

This is my first post here and I feel it's appropriate to thank all those who have had a hand contributing time and effort making TSD what it is. smile.gif

Richie

11-29-2004, 07:56 PM

Well said!

Richie

Steiner14

12-01-2004, 08:04 PM

Beside the suggestion to search for 2player battles or AI battles, i have a major problem with the FORCE BALANCE rating or even the rating system in general.

IMO it depends way too much on what the player expects. But the problem is, the player doesn't know what he can expect and so the ratings do not objectively reflect the intentions of the designer.

An example: a designer makes a scenario for very good players. Newbies or average players will usually rate it very bad, although it is exactly the challenge good players are looking for.
If you search the Depot, and have the old rating system, you will not even take a look at this battle, becasue of it's awful ratings, although it maybe could be exactly what you are looking for.

Or what about historical battles which are completely unbalanced?
Voting FORCE BALANCE would in no way be just for such scenarios and on no way reflect their intentions.

So the votings are even turning into the oposite of their task: they become misleading.

A way out of this dilemma could lie in the oposite of the actual way:
maybe we should think about a voting system, that shows intentions of the author, the expectations of the reviewer, and maybe the experience of the reviewer to get to a more valuable result.

So far only ratings of the players are discussed.
And tastes are very different. Levels of players, too. And even more what the players expect when they download a battle - although they don't even have really a clue, what they are downloading.
This obviously has to lead to misunderstandings and unsatisfied players and authors.

Usually the authors know quite good, what they created.

So how about giving the AUTHOR the possibility to rate certain aspects of the scenario?
I.e. difficulty could be a rating, that can be judged quite well by the author and the testers.
Categories could be
Winnable for newbies (against AI)
up to
Impossible for Cracks (against AI) (if you win this, you're the best of the best). ;)

If you know, you're going to play a very hard scenario, you look with completely other eyes at it, as if you expect a balanced scenario, but it is the heroic last stand of brave men.

And for H2H battles the rating could be:
Winnable for newbies (against average/good/very good player)
up to
Winnable only for Cracks (against newbies)

Important seems to me, that in the rating the level of the oponent, on which the rating is based, is clearly expressed, to get it more objective because of the different levels of the players.

That would bring much more interest on unbalanced scenarios which are really challenging.
And so it finally becomes possible, that players become happy with a draw or even a minor defeat - because it is such a hard rated scenario.

I think such a rating system would greatly broaden the bandwith of liked scenarios.
A bandwith that CM offers, but which isn't used, because every battle leaving the small path of an equal force balance has no chance in the user reviews.

Personally i'd give way more on such a difficulty rating from a respected designer, than on ten reviews of players with completely different levels and preferences.

Map design, atmosphere/fun ofcourse would stay categories to be rated from the reviewers.
Also the reviews could contain the possibility to rate the authors rating about the battle's difficulty (i.e. was way harder for me than author rated; my rating: above average; oponent: newbie) :D

Just my 2 ct.

ps: just had an idea, about a feature that could increase vastly the number of reviews.
When you have reviewed a battle, you can post your result (against a human oponent, you have to rate his level). Then, depending on the authors rating, your result and the strength of the your oponent, you receive a personal rating!
Ofcourse this rating is just a personal information and is nowhere displayed. So there's no need to tell wrong results.
Thid could be even expanded with statistics for each battle.
Example:

Your rating (based on the authors rating and your result): You're an average player.
Your rating (based on the results of other players and your result): You're a good player.

Statistics:
based on 20 results

Played as
Axis:

60% (12 players) lost this battle - 30% (6 players) with a total defeat - 20% (4 players) with a major defeat - 15% (3 players) with a minor defeat

10% reached a draw (2)

30% (6) won this battle - 25% (5) with a minor victory - 5% (1) with a total victory

Now tell me, such statistics as price for a review would not lead to more reviews!

Based on the statistics, there could be even made a ranking of the most difficult battles.

pps:
just had another idea: the Depot is marking REFERENCE BATTLES (only battles against the AI with most reviews and therefore most statistical data)
or
REFERENCE BATTLE of the month - battles which should take attention to get many reviews and results).

[ December 02, 2004, 04:17 AM: Message edited by: Steiner14 ]