Creative writing hasn't been one of the super-hyped use cases by OpenAI for the OpenAI API outside of AI Dungeon, surprisingly. For just random generation, the necessary curation can detract from the time-savings advantages. (as an aside, the API is also extremely expensive for long-form content to the point I'm not sure how the economics work for these startups even with charging monthly fees).
Generating M:tG cards is interesting to me. Can we talk about this a bit? :)
My degree dissertation was an M:tG expert system that included a hand-crafted parser and generator for (a small subset of) ability text [1]. My Masters' thesis was a grammar induction system trained on M:tG cards [2]. And I finally managed to sneak some M:tG grammar induction in my papers as a PhD student [3].
I had a quick look at the colab of your project. If I may be critical, the examples you have are ... so-so. I've seen previous attempts to generate M:tG text with neural nets, starting with the old project in M:tG Salvation that became Roborosewater and that used LSTMs if memory serves. Results have always been a mixed bag that looks promising but always falls down holes even with hand-curation. This still seems to be the case with your project.
A couple of examples (you probably know already):
−2: Put a +1/+1 counter on each other creature you control.
"Each other creature" should refer to a card with the (super) type "creature" but this card is a "planeswaker".
+1: Each opponent chooses a creature they control. If that creature was a copy of that creature, each other player reveals their hand, chooses one of those cards, then puts it into their hand, then shuffles.
Which creature was that creature a copy of?
−10: Target opponent sacrifices a creature.
−9: Each opponent sacrifices a creature.
Representing costs in a meaningful manner is a constant problem in every M:tG generator I've seen.
The problems I highlight above are not with grammaticality, which is certainly a big step forward with respect to the past. But many of the abilities still don't make a lot of sense, or don't make sense to be on the same card, or have weird costs etc. If you want your project to be something genuinely new, I have to say I don't think it's there yet.
My intuition is that it would take a lot more than language modelling to generate M:tG cards that make enough sense that it's more fun to generate them than create them yourself. I think it would be necessary to have background knowledge of the game, at least its rules, if not some concept of a metagame. It might also help to re-train standard NLP pipelines specifically for M:tG (e.g. some kind of ability-text specific anaphora resolution might help with "that creature" above).
Also, I note that the new online version of the game is capable of parsing cads as scripts in a programming language using a hand-crafted grammar rather than a machine-learned model [4] [5]. So it seems to me that the state-of-the-art for M:tG language modelling is still a hand-crafted grammar.
__________________
[1] https://github.com/stassa/Gleemin - unfortunately, doesn't run anymore after multiple changes to Prolog interepreters used to create and then port the project over.
The outputs on pageload are just a random uncurated set I ran. I never claimed smaller models like the MtG models were immune to the same curation issues as larger text generation models.
As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win.
>> The outputs on pageload are just a random uncurated set I ran. I never claimed smaller models like the MtG models were immune to the same curation issues as larger text generation models.
Are you sure? I reloaded the page once, before commenting above, but the output didn't change. I keep reloading now and the same set of cards is displayed. I reloaded with Shift-F5 to be sure ish.
>> As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win.
I don't know what an "interesting" card is, when we're talking about automatically generated cards. If a card looks too much like an existing card (e.g. if it has abilities that you can find on cards in a real-world set) then it's not very interesting. If it's an ability that hasn't been seen before but is only a variant of an existing ability ("Destroy target donkey") it's still not very interesting. To generate really "interesting" cards a generator must go beyond what's in the M:tG corpus, but not so far out that the abilities don't make sense anymore because that's not "interesting", just "random". The ones generated by your project are not far out enough to be what I'd call "interesting" but I think if you tried to make them more interesting they'd also become less coherent than they are currently (looking at the same few results I keep getting anyway).
You have to run the notebook itself by following the instructions at the top, then you can customize the output (lower temperatures will follow the rules better).
Sorry, you said on pageload, I thought you meant when the page is loaded :)
On the cards you posted to reddit:
a) "Keening Vythat", "Tombsis Satyr", "Quarpling Barrier". These are typical nonsense-nonsense names (as opposed to the fantasy-nonsense names found on M:tG cards) that are typical, in my experience, of card names generated by language models trained at the character level. M:tG fantasy-nonsense names, like "Rathi", "Kapashen", "Viashino", etc, are derived from a real-world narrative created specifically to theme the cards. A backstory. Without such a backstory, "Vythat", "Tombsis", "Quarpling" are just random strings that have no reason to be used as names. This is an important limitation of this kind of card generation, especially given the very low chance that you'll see any more "Vythat" cards (and if you did, they'd likely have a completely different function). Also "Keening Vythat" sounds like a creature, rather than a Sorcery.
So, first problem: thematic consistency of names is all over the place.
b) The text on Keening Vythat is ungrammatical:
Basic lands are basic land type in addition to its other types.
c) Seismic Mind has an interesting ability (the one that modifies its base P/T). The mana cost makes sense, actually (although I bet it'd end up at {4U} at R&D), and the effect is very Blue. Nice.
d) Tombsis Satyr also has an even more interesting ability (feeds the inner Johnny). The cost and colours are right and the Angel type is just the cherry on top. Very nice. If only it wasn't called "Tombsis Satyr"... X)
e) Quarpling Barrier's ability is more of a crippling weakness than an interesting ability. A creature without Defender would not be called a "Barrier" these days, either. Pass, I fear.
f) Trade Damage is more balanced than interesting, but it's very well balanced! It would fit very nicely in a modern tempo deck. Nice also.
So that's three nice, two not so nice and a bit of a problem with themes. Still, those are not that bad at all. Unfortunately, there's just five of them. At this point it's hard to tell whether your project is able to generate better cards than other projects - and why would it, if it's using the same tech as those projects? You can fine tune and turn more knobs, but what, fundamentally, is the innovation here?
I suggest that you add the five cards above (and/or others) to the colab of your project alongside the auto-generated ones, to have examples of the "best case" and "average case" together. That'll give a boost to the appeal of your project.
Anyway, better than I originally thought but still a ways to go.
Again, this (and my other text generation projects) are optimizing more for humor/chaos than accuracy, and there is a bolded important notice before the text emphasizing that the results may not be legal. Per the other comments on that Reddit thread, that appears to be the correct approach as the crazier-but-barely-legal cards are what resonate the best (the "bad" RoboRosewater cards were the most popular, not the good ones).
Curating only cards which are grammatically good is in itself misleading. Part of the reason I work on text generation in the first place is to illustrate its limitations.
I encourage you to run the notebook itself instead of arguing a No True Scotsman about something I never tried to assert. This notebook is not a thesis paper, just a side project about a ML model that took more time to train than to write the code for.
I'm not arguing a "No True Scotsman". I offered criticism to help you improve your work, as criticism should do. I'm sorry that you took it the wrong way.
Creative writing hasn't been one of the super-hyped use cases by OpenAI for the OpenAI API outside of AI Dungeon, surprisingly. For just random generation, the necessary curation can detract from the time-savings advantages. (as an aside, the API is also extremely expensive for long-form content to the point I'm not sure how the economics work for these startups even with charging monthly fees).
I'm more bullish on small bespoke models for a given use case, which is what I spend my time researching. (example: a ~1M parameter GPT-2 Magic: the Gathering card generator: https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZm... )