Is this the final centralized topic for this idea now?   Regardless, let's see how far we can take this idea so that Frogboy responds.

yeah i fully endorse this idea.  i really like the randomness and didn't want to give it up, but this way it makes more sense and is still random(its even MORE random mwahahaha!).

I honestly like this idea especially at the beginning. But I wonder toward the late game when you really want to learn a specific tech because you know it will help you do x,y,and z and your minions keep coming back to with techs you don't want.

Alright, I've finished running around the park at night with a light up frisbee, time for me to take a crack at this.

Beric01, it seemed like a good idea at the time to split the threads between name brainstorming and actual functionality.  Maybe that's a bit overkill, but oh well .

If there's one thing this subcategory system reminds me of, it's the undirected research of Alpha Centauri.  This was interesting, but I don't remember how it felt to play undirected research.  I do remember turning on directed research for most of my games, but I'm a horrible power-gamer .

I've got silly name titles for the subsections, but I basically read down the thread and tried to digest each person's ideas and answer questions as best as I could brainstorm myself.

What's in a Color (Kantok):

Right now, color rarity means a very specific thing: the chance a technology is available when you breakthrough.  Keep in mind that currently red techs hold something like a rarity of between 20 - 30, which means that if you breakthrough 10 times after you've satisfied prereqs,  you are guaranteed to have the tech.  Most of the reds right now do not break the game wide open right now, and honestly I don't think they should.  I do think that they should be valuable and useful, and perhaps characteristic.

Valuable: The technology opens up new inventions which are more potent than similar inventions available in that subcategory.

Useful: The technology is not terribly situational for its category.  For instance, I wouldn't make Beekeeping a red technology with this understanding of color.  An alternative reading of this sentiment: that the technology will be usable for most people who are given the opportunity to discover it.

Characteristic: Defining.  We know the Japanese for their swordsmiths, the English for their ships, the French for their universities etc. - this is what I mean by characteristic.

I think you have Control Issues (Kantok):

The easiest way to take the sting out of the control issues is to make sure the player is most always pleasantly surprised when they breakthrough and get their technology - or at least, that their minions are researching in a competent manner to their wishes. 

I think this will require some under the hood checks, eg. not to give out farming unless the player has control of grains, not to give orchards unless they have fruit, etc.

We Can Rebuild Him, Faster, Stronger, Better than Before (Kantok):

XML is incredibly flexible, and when combined with python scripting you can do some pretty crazy things.  For instance, I have an in-house CIV IV mod right now that adds nand & nor prerequisites, hides technologies you can't currently research, and colors the background of a technology based on its cost.  I'm not a super coder or anything like that, but with some careful thinking, changes like this aren't that big of a deal.

Basically, give us your best ideas, and worry about implementation later!

Note: The Nand & Nor Prerequisites Mod required some .DLL modifications (C++), mostly to add additional XML Tags.

You're Just One Unlucky B*stard, Aren't You (Kantok):

Here is my first truly radical idea: we've bounced around the concept on these forums about gray technologies, techs which you have so far surpassed that you automatically discover them with your breakthroughs.  I've started thinking, perhaps we're thinking of this wrong.  Instead of a tech becoming gray after a period of time, why not have different classes of breakthroughs!  You would pop-up with a message to the player saying: "Congratulations, your scholars have finished their research in [subcategory name] and have achieved [type of breakthrough] rediscovering the following technologies:[][][]", yeah, you read that right, technologie(S).

Dramatic Success: 1 Red Technology and 1 Green/Yellow Technology

Success: 1 Yellow/Green Technology and 1 Green Technology

Moderate Success: 2 Green Technologies

This is pretty huge, and if we went with it, we would definitely need to start breaking some of the technologies apart and deciding what really is green versus yellow, versus red.  I imagine that I would structure the percentage chances in favor of moderate success in a 60:30:10 ratio.  However, this also means we can have things effect your breakthrough types, like sovereign traits and inventors!

Game Over Dude (Iswallie):

The late game is where the system has a hard time coping, though I think a system which uses typed breakthroughs would handle better.  Another way to deal with the late game setup (where you really only want significant technologies, since you've covered the basics which you want already) is to use the gray technology idea which has been bouncing around for a while.

One, Two, Three? Or Two, One, Three? (sjwt):

This works in a similar way to the pulse system I tried to explain in the Technology System Tweak thread, you should read the paper, and not just because I wrote it! .  Out of order technology acquisition is always a little strange: after all, if we've discovered lighter swords, why should my researchers ever waste time with lesser technologies.  I solved this in the pulse system by isolating groups of technologies (Pulses) and only having out of order discovery occur in each pulse.

Now, here's a fun question: How many subcategories is too many?  The degenerate case is to have a subcategory for each technology, but how many subcategories should we be comfortable with.  I think no more than five, to have a parallelism with the technology categories themselves.

Unlocking categories with technology sounds interesting, but then we run into our tree problems again...

Summarizing the General Philosophy:

Research with subcategories is supposed to reflect the player telling his research minions to solve a particular problem, and his minions coming back with a solution.  It may not be the best solution, but it will help solve the problem the player asked them to solve.  The system falls apart if it doesn't, because players become frustrated at their lack of control, or at least, I would.

Therefore, while balancing the tracks, we need to keep in mind that they all need to be valid solutions to the problem the player claims to want solved.

Redefining Color:

Currently, technologies are organized along lines, or tracks (the word used in the XML): linear prerequisite trees.  The player advances along the track, with his researchers choosing tracks based on some reasoning, attempting to solve the problems the player identifies.

I say, that for each step along the line, there should exist different versions of each technology: Green, Yellow, and Red.  Green is the default, Yellow is a little better, and Red is a true marvel.  Which version the player gets should be dependent upon the type of breakthrough they have achieved.

Players following the same tech pattern then, are going to be slightly better and worse at different things.

Defining Your Civilization:

This brings in another opportunity, to tie in the academies and typed research points that were mentioned a while back (a week or so, give or take a few days).  Now, instead of academies giving more research towards a particular category, they would instead increase the chance of better breakthroughs in their associated categories.

Okay now that I've had my coffee and had time to ponder this...

What's in a color?:  I think I was unclear in what I meant by red techs being society changing.  I don't want them to be drastic, hard to balance things.  I see them more as the epitome of their field.  In the farming category, red techs might be irrigation or crop rotation.  For siege weapons/ballistics it may be the trebuchet or ballista.  Something powerful in such a way that they can change how you handle whatever category they relate to (farming or siege warfare, in my examples). 

I especially like the idea of characteristic red tech's, but that adds balance issues and now you need to come up with a way (not overly hard from a conceptual standpoint, but it's more change) for player created societies to define their characteristics (My people are a seafaring race, for instance) and to have that come into play in little spots throughout the tech tree.  If you don't have it come into play occasionally throughout the tech tree it becomes a superficial choice and basically turns into Civ4's special unit.

Control Issues:  My concern here is that some players just don't like the idea of not having a nicely mapped out tech tree.  The idea of subcategories and getting back an appropriate tech that the game chooses for you further exacerbates this.  By eliminating the "Tech Discover, pick from this list!" that we have now, you're moving even further away from tech trees and the ability to map out where you're going.  Personally, I like the idea of only having Macro control over your research (that's why I proposed the idea) but I know many players won't. 

I don't agree that you would need to be overly careful to make sure that each discovery is directly relevant to the player right at this moment.  That goes counter to the idea that your researchers are looking to develop new technologies or REDISCOVER old ones.  By not guaranteeing that each tech is immediately relevant you also eliminate Icepick's concern that if you don't have a mine you never get mining and hence all techs that are based on mining are now cut off to you until you get a mine. 

You're Just One Unlucky B*stard, Aren't You:  I don't like the idea of different classes of breakthroughs, at least not as presented here.  I think a simpler mechanism would be to "gray out", as you say, certain techs that you've clearly surpassed and that any grayed out tech has a chance to be auto-discovered when you're doing research in its category.  IE. Once you're "Warfare -> Weapons" level is high enough, daggers becomes gray (all of your research has succeed in bow and wooden weapons in this example) and any techs that have "daggers" are their direct prereq become available to be rewarded to you.  If one of those techs are discovered, you're researchers "discovered daggers" in the process of figuring out how that next tech and you're presented with both.  "Sire, we discovered how to make short swords!  In the process we also developed the ability to make really tiny ones called daggers!"  This removes the chance for the game's rng to penalize the players through a run of bad luck. 

This also helps the late game "find specific tech" problem that Iswallie mentioned.  The more advanced of a warfare society you become, the more likely you are to have nailed all the early basic techs (because they were either discovered early or auto-discovered).  If there are dependencies between techs (even across subcategories) this means that as you auto-discover the basics through your advanced research you're also satisfying the dependencies and opening up further research. 

I think the idea that you get to late game and all the sudden want "that one specific tech" defeats the whole point of the system.  How is it okay to want that one specific tech in the late game, but not in the early game?  I think once you get to the end of the tech tree in a given subcategory (not that we see the tree, but the game knows it) you're chance of researching specific techs (as opposed to the generic "Melee Weapons II, Melee Weapons III that will be at the end if you want to keep incrementally improving your ability in a given area) should increase so that within a few research cycles you've researched all the available techs and all that is left is the incremental improvement techs. 

This graying out also eliminates the 1,2,3 or 2,1,3 technology problem because once 1 is grayed out and you discover 2, you're given 1 as a "hey we worked this (1) out in the process of discovering (2)".  That leaves it up to the player to decide if they have a use for tech (1) or not and if not, they just let it sit and go about using (2).

I'll post more response in another post in a bit.  After I have another cup of coffee. 

Edits:  Should have had the second cup of coffee first.

Characteristic:

With typed breakthroughs, you could push higher chances of better breakthroughs in particular subcategories.  You'd want to shift the odds in such a manner that the player notices the change.  Eg. my faction trait is seafaring, so when I'm researching in the seafaring subcategory of civilization, I am more likely to have moderate and dramatic successes.

Another way to do typed breakthroughs are to only have one technology at a time, and then we're just adding names to the technology color categories.  I think the X + Green system is better for the stronger breakthroughs because it emphasizes that you are making both progress and getting extra benefits, either from luck, characteristics, or because of other factors which the player has control over (like the academies idea).

We can also fold in a gray technology system here by adding 1 gray tech to each type of breakthrough.  The end result is that your researchers come back to you, and instead of saying: "We discovered X", they can say "We looking into [Category] and figured out how to do X,Y,Z".

Control Issues:

Let's set aside people's uneasiness with having only macro control of their research for a brief moment, and come back to it later.

RNG Management:

I think the biggest problem with the gray system is that people haven't really acted that keen on an endgame technology setup which converges for each category - such systems work, but they're a little bland (CIV).  Should it converge towards similar results (eg. two warfare specialized factions will have access to most all of the weapons and be specialized in one or two each)?  I think if we keep the red technology flavorful and out of the prerequisite line, we'll have enough differentiation that it won't be a problem.