Thank you for this thoughtful and detailed reply. I have read about Arithmetic Coding and Data Compression. I haven't read about Solomonoff-Induction but I will definitely read up on it.
I think another way of thinking about my original comment would be to imagine that you had an image of the Mandelbrot Set up on your monitor (assume you have drilled down a few times). If you knew the parameters that had generated that view you could write a very concise description suitable for transmission over a network. If you have forgotten the params and had to make due with a screen print saved to jpeg it would appear to contain more 'information'. Not only would the algorithmic version be shorter (so long as you hadn't drilled down to the point at which the numbers representing the view port had millions of degrees of precision) but you would be able to keep zooming in to view arbitrary detail. That is something you wouldn't be able to do with the jpeg. I guess what I'm driving at is that I don't understand how Information Theory is able to account for things like fractals without saying that either the Mandelbrot Set is random or that it contains infinite information. Am I looking at this problem correctly?
> I guess what I'm driving at is that I don't understand how Information Theory is able to account for things like fractals without saying that either the Mandelbrot Set is random or that it contains infinite information. Am I looking at this problem correctly?
This is (I think) the "coastline paradox": coast lines have different lengths depending on scale. Length grows with resolution, so in a sense the length of a coastline can be infinite.
I think the escape hatch is that the information conveyed is not the points on the screen, but the rule on how to generate those points. Going back to my other post this means that the city builder conveys no information about a particular city map.
> I think the escape hatch is that the information conveyed is not the points on the screen, but the rule on how to generate those points. Going back to my other post this means that the city builder conveys no information about a particular city map.
Yep. The amount of information conveyed can be considered 'equivalent' to the length of the shortest algorithm used to get from one string to the other.
The field that is concerned about thinking about this problem in a disciplined manner is algorithmic information theory:
I think another way of thinking about my original comment would be to imagine that you had an image of the Mandelbrot Set up on your monitor (assume you have drilled down a few times). If you knew the parameters that had generated that view you could write a very concise description suitable for transmission over a network. If you have forgotten the params and had to make due with a screen print saved to jpeg it would appear to contain more 'information'. Not only would the algorithmic version be shorter (so long as you hadn't drilled down to the point at which the numbers representing the view port had millions of degrees of precision) but you would be able to keep zooming in to view arbitrary detail. That is something you wouldn't be able to do with the jpeg. I guess what I'm driving at is that I don't understand how Information Theory is able to account for things like fractals without saying that either the Mandelbrot Set is random or that it contains infinite information. Am I looking at this problem correctly?