This is an archive of a previous version of Sodarace.

forum  |  »» sodaplay

»» forums  »» artificial intelligence forum subject: challenges 13 replies on 1 page. most recent reply: 22-May-06 00:37 by BOBZUDA

»» back to topic list

This topic has 13 replies on 1 page

»» previous topic   »» next topic

sml »» models »» homepage challenges   posted: 03-May-02 13:56   »»  I think Sodarace is a great idea. I've enjoyed seeing many of the great sodaplay models sent in by the model makers, but I've always thought: how about some objective challenges that we can use to pit man against machine, to see which model can sprint the fastest or throw a Javelin the furthest. Or how about Sodafight - where two models do battle until the death - this could be based on simple rules such as a mass destroying an opponents spring on any contact. Watch the models gradually obliterate each other in a war of attrition. Perhaps models will evolve fancy tricks - maybe it turns out that springs break under too much extension - hence possibly releasing masses as projectiles to aim at the opponent without getting in harm's way - but a risky strategy - like tearing off your own arm to throw at somebody! The scope for machine learning algorithms - and challenges for human designers here is immense. I can imagine a first sodarace phase based on the current models - this will be a great success - but sooner or later, people will demand sensors and processing elements (e.g. simple neurons). These will have to be designed in carefully to ensure that the simulation remains tractable. There are plenty of other design challenges that need addressing - like specifying standard encodings for constructor kits, and model representations - I'd propose simple XML based standards - these are naturally extensible, and will allow competitors writing in any language to download toolkits and upload their designed models. (re: constructor kit I imagine that for each challenge we might get given a limited set of parts and the challenge is to find the optimum configuration of these parts). Also at issue are the simulation enviroments - researchers will need to be able to download their own copies of these - e.g. I might need a lot of CPU power to evolve my Soda models into a highly fit state before sending them up to compete with other models. That's probably enough for a first posting - I look forward to follow-up discussions on this. ps. Nearly forgot to mention - thanks very much to Ed and Sodarace for sponsoring the WCCI 2002 competitions - not too late to enter and win a Sodarace tee-shirt! - see: http://ace.essex.ac.uk/algoval/wcci/CompetitionIndex.html for more details.

ed »» models »» homepage Re: challenges   posted: 03-May-02 15:41   »»  great to hear from you, very glad that you like the sodarace idea, lets hope it will run and run! > - but sooner or later, people will demand sensors and > processing elements (e.g. > simple neurons). These will have to be designed in > carefully to ensure that the simulation remains > tractable. Sensors and nervous systems are very interesting and have for example been evovled to great effect at Framesticks ( http://www.frams.alife.pl/ ) and in the work of Karl Sims ( http://www.genarts.com/karl/ ). I'm rather cautios about going down that route at sodarace for the following reason: If you've tried manually making walking sodaconstructions you'll probably know that it's already waaay difficult enough even without the challenge of designing nervous systems as well as bodies. Sodarace has got to be accessible to both AI's and Humans for it to be an interesting comparision of human and machine skill and creativity, so I'm a big fan of keeping the building blocks as simple as possible. We should float this idea in the model maker forum maybe to see what experienced sodaconstructors think. > I'd propose simple XML based standards - these > are naturally extensible, and will allow competitors > writing in any language to download > toolkits and upload their designed models. Yes indeed, XML looks like it could be a key factor in building sodarace to be as open and inclusive as possible. A major project for us in the AI forum will be coming up with a sodarace architecture that promotes a modular approach that will be responsive to real research needs. I'm looking forward to seeing how this train of thought develops.

sherwoac »» models Re: challenges   posted: 09-May-02 12:34   »»  XML: I also think an XML definition is a good idea. Non-visual: It might be useful to have a non-visual version of the main applet that allows the following: - Loading of XML animals - Selection from the various courses - runs a 'time-trial' of the animal traversing the course, without displaying it, and returns the time taken to for the loaded animal to complete the course. The basic problem then becomes minimizing the time returned by the applet. The benefit of it being non-visual means it would run more quickly and aid the automated breading of animals.

ed »» models »» homepage Re: challenges   posted: 09-May-02 13:28   »»  > The benefit of it being non-visual means it would run > more quickly and aid the automated breading of > animals. yes indeed, many learning algorithms are likely to be very processor intensive, requireing race results to be generated for large numbers of models. The process of sumbitting models and obtaining results will certainly have to be able to be full automated via the API. So the software architecture needs to be both open AND efficient for it to be a useful task domain for exploring the realtive merits of various genitic algorithms for example. Has anyone seen other relavent examples of algorithm evaluation on the net that might have lessons for the sodarace design?

TimT »» models »» homepage Re: challenges   posted: 09-May-02 13:36   »»  I agree that providing an XML standard for specifying a creature (at a low level - i.e. the dimensions and positions of individual springs and joints) would be a good way to go about this. This leaves people completely free to develop the creatures however they like (using AI techniques or by hand) - the only requirement is that any technique generates an XML file as its final output. > Sensors and nervous systems are very interesting and > have for example been evovled to great effect at > Framesticks ( http://www.frams.alife.pl/ ) I'm not sure if this is really true. There are some nice Framsticks creatures that have been designed by hand (i.e. most of the movies on the website), but if you actually look at the ones that have been _evolved_, the results aren't quite as impressive. But despite that, I would encourage the developers to add some sensors. It may be harder to design creatures by hand to use sensors, but for people designing creatures using learning techniques, sensors can often make the task in hand somewhat easier to achieve (e.g. walking can be easier with orientation sensors in the limbs).

ed »» models »» homepage Re: challenges   posted: 09-May-02 14:00   »»  > I agree that providing an XML standard for specifying > a creature (at a low level - i.e. the dimensions and > positions of individual springs and joints) would be > a good way to go about this. This leaves people > completely free to develop the creatures however they > like (using AI techniques or by hand) - the only > requirement is that any technique generates an XML > file as its final output. Yep, having it as accessible as this could make race entrants completely platform, algorithm and language independant, which is great. It does open a can of worms though: should we find a way to validate the inner workings of AI entrants in order to make meaninful comparissons between entrants [both human and AI]? I think there are some tricky issues for us to grapple with in this area, do reasearchers want to reveal the workings of their algorithms for example or are there intellectual property issues to protect, and do we draw a line between hand-crafted models and the results of machine learning algorithms?

guest: Alex Bailey Re: challenges   posted: 14-May-02 13:30   »»  > It does open a can of worms though: should we find a > way to validate the inner workings of AI entrants in > order to make meaninful comparissons between entrants > [both human and AI]? I think there are some tricky > issues for us to grapple with in this area, do > reasearchers want to reveal the workings of their > algorithms for example or are there intellectual > property issues to protect, and do we draw a line > between hand-crafted models and the results of > machine learning algorithms? You have highlighted an issue which crops up in many computer science competitions, since an entry may be entirely computer generated, or may have had some manual tweaking aswell. It might be difficult to tell if someone has designed a creature using a Genetic Algorithm, say, and then fine-tuned the final result before entering in the competition. This could be considered cheating. On the other hand most AI programs will need a certain amount of tweaking of parameters, and initialisation, perhaps using manually designed building blocks. Is this cheating? It may be the case that the only way to do it would be to have a free for all, and let the best creature win, regardless of the design process. All entrants will be constrained by the same physical simulator, so that's fair. Whether the designer has to disclose the technique used to design the creature or not is a different matter. I expect most designers would want to show off their methods to some extent, but they needn't give it all away if they are concerned about intellectual property.

ed »» models »» homepage Re: challenges: validate or debate?   posted: 14-May-02 15:05   »»  > It may be the case that the only way to do it would > be to have a free for all, and let the best creature > win, regardless of the design process. mm, I'm increasingly sympathetic to this approach Alex. In a recent conversation with Simon (sml, first posting in this thread) we leaned towards a similar "whatever works best IS best" approach to allowing as many entrants as possible. So, rather than trying to encode some ingeneous kind of algorithm validation scheme we can instead handle these issues to some extent through the social activity around the races [eg, discussion in this forum], where researchers will be free to promote or defend the relative merits of their diverse methods. I'd be interested if you've got any references of how other computer science competitions have handled this hot topic.

guest: Alex Bailey Re: challenges: validate or debate?   posted: 16-May-02 14:52   »»  I was specifically thinking of an information retrieval conference where a certain technique could use a dictionary learned from text, or by using a hand-built dictionary. There was some controversy since some people learnt their dictionary from the text supplied as part of the competition, and others were judged as having an unfair advantage by using a dictionary that was not part of the competition data. Unfortunately I can't remember enough to find a reference. The conference is called TREC, but they runs several distinct tracks, each with it's own 'competition'. In any case each member is obliged to publish a paper detailing their technique. Most machine learning conferences / competitions supply a dataset to train a neural network, say, and then they give a test set without the results. You have to send in your results on the test set, and they are ranked in order of accuracy by the organisers who hold the correct results. I remember ERUDIT as a classification competition. I guess this doesn't apply to sodarace since there is no training data as such. The problem is defined by the sodaconstructor API and simulator. You just run the simulator and the best creature wins. There must be competitions for reinforcement learning techniques which might be more similar. There was an algorithm called TD-Gammon which won a reinforcement learning competition to learn to play backgammon. Anyway, in response to your topic, the techniques are published as papers and debated at conferences. I've not heard of methods for automatically validating algorithms, I'm not really sure what you mean.

guest: Alex Bailey Re: challenges: validate or debate?   posted: 16-May-02 15:09   »»  I've just looked up Simon from the first post. I'd say he's certainly better qualified on this subject than I am. Out of interest, how many entrants would you expect, and is there going to be a deadline and one almighty race with heats etc., or an on-going league table with daily challengers, more like a squash ladder?

ed »» models »» homepage Re: challenges: validate or debate?   posted: 16-May-02 15:54   »»  > Out of interest, how many entrants would > you expect, and is there going to be a deadline and > one almighty race with heats etc., or an on-going > league table with daily challengers, more like a > squash ladder? Ongoing races is definetly the way to go I think. I'd like anyone to be able to enter at any time to encourage as much participation as possible. This means it is very likely that sodarace will operate as a series of user-initiated time trials rather than a synchronous race. An ongoing competition will be manifested in the league tables where the results of these time trails will be ranked, encouraging participants to refine their entries and re-enter the race to scale the leader board.

abclark »» models »» homepage Re: challenges: validate or debate?   posted: 17-May-02 20:07   »»  what.

Lectvay »» models »» homepage Re: challenges   posted: 17-May-02 23:40   »»  Going off of that idea ed... Maybe you could officially start races, and they would last a few weeks or so. Then, people would send in models as much as they wanted, and there would be a scores list, where everyones time is listed, but their models are not shown. It would be an ongoing competition for those two weeks. Then, after the race is officially over, the final scores list would be shown, and the models would finally be revealed. This way, privacy would be kept for the duration of the race to keep it competitive, but then afterward everyone ideas would be visible.

BOBZUDA »» models Re: challenges   posted: 22-May-06 00:37   »»  SHAZAM!!!!! Lol i have wanted to do that for so long... wait... i forgot to read what this topic is all about... ...processing... okay ive read this whole page and i agree that what ed is saying and what the author is saying are both interresting, and i think that there should be done about Sodafight. i think it would be a great idea to do, and it can probably be accomplished with the freemass barsprings, if the tweaks are to be worked out. ()()()()()(EGSTOD)()()()()()

This topic has 13 replies on 1 page

»» previous topic   »» next topic   »» back to topic list  »» top of the page

PLEASE READ THE FORUM GUIDELINES AND ALWAYS PREVIEW TO CHECK MESSAGES BEFORE POSTING...
...Help keep our forums creative and constructive. Thank you.

»» forum home