The goal of my research has been to improve Tool X so it can be used in practical situations. While I occasionally worry that this goal is too engineering based (implementation rather than research), there has been some science in the ways I try and improve Tool X, and in the ways I analyse the results. One of the ways that other people have improved their tools that do similar things to Tool X is to add Tool A into the system somehow. And so, I looked at various ways to integrate Tool A with Tool X, ran a whole lot of experiments and analysed the results. Major result: Tool A does help, but not to the same extent that others have seen.
Now, Tool X produces something slightly different to other similar tools. Something that appears to be both more difficult to produce, but also potentially more useful. (Both these claims are hard to quantify though.) It also has some differences in mechanism that might mean that the sort of assistance Tool A can provide is not so necessary in our case. All this means it is difficult to directly compare with others. In any case, I wrote up the results of my experiments and sent them off to one of the mentors I was meeting a few weeks back. At the conference, we discussed these results, and he pointed out that the standalone results I was getting with Tool A were not state of the art. He mentioned a more intelligent variant of Tool A, which I would easily be able to run now I had the infrastructure set up, and which he was sure would give better (Tool A) results since it had been used in passing in other experiments on the same sort of data. So I tried his version of Tool A when I got home, and it did slightly worse (in terms of Tool A results). I then started doing some more focussed reading on Tool A variants and tried an even more intelligent variant which gives state of the art results for others (and is completely open source) and it gives significantly worse results.
So now my quandary. Tool A was never my focus. My old results are still valid: I get an improvement using Tool A with Tool X contrasted with using Tool X alone. I can show the maximum improvement I would get in Tool X if Tool A gave perfect results. But my recent reading and exploring has suggested that the sort of data I am working on has certain properties that indicate that Tool A could be used in a different manner to that of everyone else. Now I am trying to decide if I should put more time and effort into examining how data can be fed into and extracted from Tool A in different ways. That would change my thesis focus from "ways to improve Tool X" to "how Tool A improves Tool X, and why that is different to how Tool A improves other things like Tool X". I like the second option because it is more focussed, and it is more science than engineering/implementation. And there are interesting questions involved. On the other hand, it is quite late in the game to be changing focus like that.
The new focus would be more related to the Cupcake-ology side of the field, which is exciting because I think more Cupcake-ology results should be used in our field, but also daunting because my background is strictly Computer Science. I have taken only a couple of Cupcake oriented courses, and most of the Cupcake information I have picked up has been through experimenting (as a computer scientist) with Cupcake data, and though talking with Cupcake-ologists who have moved to our field. I feel a bit imposter-ish trying to make even general claims about Cupcake theories in front of others who have actually studied the field.
So now I am trying to decide how far to take this. I will need to do more with Tool A anyway, since I agree that I need to be closer to state of the art to make any real claims. But do I try exploring these different ways of using Tool A, and why that works better in Tool X and what that tells us about different Cupcake theories, or do I just put in a reasonable amount of optimisation work and show the Tool X upper bound that a perfect Tool A would give? It actually wouldn't change the structure of my thesis much in either case, but it would change the slant of the writing and probably the conclusions I am trying to draw. In some ways (given time restrictions) it would remove some breadth from my thesis, but add some depth (in an unexpected place).
Anyhow, I really need to figure this out this week. Or at least a plan that dictates how long I try to do any particular thing, and what my fallback options are. It would be much easier to write my thesis if I could settle the direction...