[#!: full-line executeEmbeds,  phraseValueList = (("fout  fout)("backtrack  backtrack))
[#!: path_insertIn_fHand					(link  d_webWork  'fin Head_one.html')		fout  ;  
<TITLE>Wilhelm Abel, David Fischer - The Great Pricing Waves 1200-1990 AD</TITLE>    
[#!: path_insertIn_fHand					(link  d_webWork  'fin Head_two.html')		fout  ;  
[#!: path_executeEmbedsInsertIn_fHand  (link  d_webWork  'Menu.html')  phraseValueList  ;  
[#!: path_executeEmbedsInsertIn_fHand  (link  d_webWork  'Menu projects.html')  phraseValueList  ;  
[#!: path_insertIn_fHand					(link  d_webWork  'normalStatus.html'	)  fout  ;  


<H2>Wilhelm Abel, David Fischer - The Great Pricing Waves 1200-1990 AD</h2>
<div class="a">
<I>"...   History never repeats itself, but it rhymes.   ..."</i> (?John Maynard Keynes?)
</div>


<h3>Table of Contents</h3>
<UL>
	<LI><A HREF="#Key [results, comments]">Key [results, comments]</a>
	<LI><A HREF="#How can the Great Pricing Waves be correlated with "dead" system?">How can the Great Pricing Waves be correlated with "dead" system?</a>
	<LI><A HREF="#Ratio of actual to semi-log detrended data : [advantages, disadvantages]">Ratio of actual to semi-log detrended data : [advantages, disadvantages]</a>
	<LI><A HREF="#Future related work">Future potential work</a>
	<LI><A HREF="#Comparison of [TradingView, Yahoo finance] data">Comparison of [TradingView, Yahoo finance] data</a>
	<LI><A HREF="#[data, software] cart [description, links]">[data, software] cart [description, links]</a>
</ul>
<BR>
NOTE :  The model here is DEFINITELY NOT suitable for application to [trade, invest]ing! It's far too [early, incomplete, immature, erroneous]!  See <A HREF="http://www.uct-news.com/">Steven Puetz's http://www.uct-news.com/</a> for the investment side of the UWS. <BR>
<BR>


<HR><A id="Model results"> </a>
<h3>Model results</h3>


<A id="SP500 & DJIA indexes back to 1872"> </a>
<H4>"SP500 & DJIA indexes back to 1872"</h4>

Linear semi-log segments [1872-1926, 1926-2020]<BR>
<BR>
Fibonnaci levels versus log-levels <BR>
Larger Fibonnaci's at greater timescales<BR>
<BR>
How do I get negative numbers from a log transformation of the data?<BR>
<BR>
<BR>


<HR><A id="What's [wrong with, missing from] the current model?"> </a>
<h3>What's [wrong with, missing from] the current model?</h3>

<UL>
	<LI>The data sets : 
		<UL>
			<LI>It is often a [huge, perennial, misleading] mistake to represent real data as smooth curves.  [Averaging, filtering] ruins data to some extent.  So I keep doing it anyways...  snspot cycle is a perfect example
			<LI>So far, I have only used one source (David Fischer), but there is a huge amount of global information on this subject of pricver history.  As Fischer and othehave stated, most historical data has been pricing!
			<LI>I had really wanted to extend back to ancient primary civilisations [Sumer-Babylon-Akkadian, Harrapan, Egyptian], secondary [China, ...], tertiary [Persian, Greece, Roman, Omec, etc]. Oh well.
			<LI>Data [quality, resolution] is variable over time 
			<LI>A metric for price over time is always problematic
			<LI>Data processing probably involved [averaging, smoothing, filtering, regional biases], and I have not looked into the sources
			<LI>There are huge transitions in the [nature, driving forces] of economy.  First of all, technology, then globalisation
			<LI>I have selected only a subset of the [dimensions, regions] of the datasets (pricing mostly of agricultural goods, not [wages, rents, land prices, etc])
			<LI>The modern context has chaano much, it sems to me that the price [revolution, equilibrium] context is not so useful, even though it's go keep in mind.  Eventually, we may hit the wall, but with personal choices radically changing families, it is unclear if the dominant population driver will play the same role, even though it still seems to be a major variable?
		</ul>
	<LI>[photos, digitization] of graphs from book 
		<UL>
			<LI>Cellphone photos were distorted by curvature of pages of the open book
			<LI>"peak shaving" (high & low) occurred with the digitization process, and I did NOT manually correct this!  (it is simple to do)
			<LI> 
		</ul>
	<LI>Simple linear regression of pricing periods
		<UL>
			<LI>The time-boundaries for pricing [revolutions, equilibria] are a fuzzy concept to begin with, and a bit artificial.  I did NOT try to [improve, optimize] this, as it's a red herring at this very early stage of this project.  One fun case is my use of 1940 as a time-boundary instead of 1926.  Food for thought, even thous it looks like a violation.
			<LI>Several time periods have [clear, strong] curvature  is not accomodated by linear regression.  At this time, I have zero interest in playing with non-linear [transitions, periods], as there are far mre important issues at hand.  I would far rather put time into fractional order calculus and spiking neural networks.  
			<LI>The discontinuities across time-boundaries is very clear with the plot of grain prices, as I used the time-boundaries for consumables prices.  It would be easy to adapt time-boundaries for each dataset, but that also opens up issues.
			<LI> 	
			<LI> 	
			<LI> 	
		</ul>
	<LI> 
		<UL>
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
		</ul>
	<LI> 
		<UL>
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
		</ul>
	<LI> Multi-fractal (Benoit Mandelbrot) and Fractional Order Calculus (Gottfried Liebniz) approaches
		<UL>
			<LI>Oh well, no time for this now...
			<LI>non-linear frequency splitters etc etc
			<LI> 	
			<LI> 	
			<LI> 	
			<LI> 	
		</ul>
</ul>

<BR>


<A id="Are [political, economic, philosophical, psychological, sociological] theories unimportant?"> </a>
<H4>Are [political, economic, philosophical, psychological, sociological] theories unimportant?</h4>

The easy answer is "YES", as these seem to pre-occupy everybody, and everybody seems to have their own ideas (which are almost invariably other people's ideas, which are almost invariably of origins in antiquity) to impose on others.<BR>
<BR>
But a justifiable interpretation of the price revolutions, and the analysis of BOTH [Alfredo Pareto, Benoit Mandelbrot] that wealth distributes are the same across societies (slopes can differ), is that our "brilliant ideas count for far less than our egos tell us.  This seems be a particular problem with intellectuals, perhaps because of their very [nature, priviledged positions].<BR>
<BR>
While ideas seem to be important, it's easy to forget that they also can cause massive problems, and there is little indication that we can think this through reliably. <BR>
<BR>
I almost suspect that with progress, machines may better derive models and concepts than humans, which might be better based on data rather than "conclusions-driven bias" of [fragile, incomplete] human intellect?  In many cases, they could hardly do worse.<BR>
<BR>


<HR><A id="And for the future?"> </a>
<h3>And for the future?</h3>

Picking up a point from the section below "Linear regression of price [revolution, equilibrium] segments" : <BR>
We don't know [how, when] the "modern price revolution" will end, so I've merely used the last data-point.  It's probably going to be bad, but as Fischer states, the impacts on people of crashes from price revolutions have become FAR less traumatic over the last eight hundred years.  I suspect that this is due almost entirely to technology and changes from local-to-global trade.  These have generated increasing "better than the pharohs" wealth, far greater stability.  
<UL>
	<LI>Perhaps our ego-centric assumed [business, social, political, rural-to-urban] brilliance is merely [driven, afforded] by those factors?
	<LI>Will the "modern price revolution" continue as long as technology continues to advance in a manner that generates greater wealth or is there some kind of [resource, environmental, over-population] limit to that?
	<LI>Will under-population become a main issue soon, as the very nature of families change with the [desire, aspiration]s of the population?
	<LI>Will "hybrid" [human, machine] living, plus increasing levels of the "economically competitive cognitive level" also [drive, force] changes in human nature?
	<LI>It seems that the largest modern [mythology, rigion, science] is socialism, and creating a "better" human is at the origins of much socialist thinks (including eugenics).
<BR>
<BR>


<HR><A id="???"> </a>
<h3>???</h3>

<BR>


<HR><A id="Was 2007-2009 the modern equivalent to the 1929 crash?"> </a>
<h3>Was 2007-2009 the modern equivalent to the 1929 crash?</h3>
<BR>
Was 2007-2009 the moderen equivalent to the 1929 crash, less the socialist abuse of the free market?
??MacKay?? reference

??MacKay??'s commments are certainly unconventional, but merit closer attention. 
Key points : 
<UL>
	<LI> 
	<LI> 	
	<LI> 	
	<LI> 	
	<LI> 	
	<LI> 	
	<LI> 	
</ul>
<BR>



<HR><A id="How can the Great Pricing Waves be correlated with 'dead' systems?"> </a>
<h3>How can the Great Pricing Waves be correlated with 'dead' system?</h3>

<BR>

<HR><A id="Inspirations"> </a>
<h3>Inspirations</h3>

Inspired by many historians and financial market analysts, most notably but not limited to (in order of appearance on the world stage) : <BR>
Ibn Khaldun "The Muqquadimmah"<BR>
?? Tijchevski? - 10 year cycles & economic activities<BR>
?? Kondriatieff - ~50 y (Mayan) super-cycles<BR>
?? Elliot's waves<BR>
Arnold J. Toynbee "A Study of History" <BR>
	<LI> (the full 10-12 volumes, I was missing one or two, now I have only the abridged two-volume set</li><BR>
Ivanka Charvova - best concept I've seen of solar activity over 5-10 ky<BR>
Steven Yaskell and Willie Soon "Grand phases on the Sun" - wonderful descriptions of the Maunder grand solar  minimum and many scientists and others<BR>
Harry Dent - the most successful 10-40 year [economic, financial] forecaster that I am aware of<BR>
Robert Prechter's Socionomics, the first quantitative Sociology<BR>
Peter Fischers "The Great Wave" -  I forget which author (probably Stephen Puetz who used some of the data) mentioned Fischer's book, but it is a real gem, built on the meticulous work of numerous historians.  I consider it to be one of the great historical works. <BR>
Stephen Puetz, and especially co-authors [Glen Borchardt, Andreas Prokoph, ??], <BR>
	<LI> and the system of 20+ Mayan calendars</li><BR>
Ray Dalio of ?BlackRock" Hedge fund "??" - nice historical analysis from the point of view of a financial industry leader<BR>
<BR>
<BR>


<HR><A id="References"> </a>
<h3>References</h3>

David Fischer does a superbv job of listing and discussing a huge number of references on historical pricingg and the issues that arise!!!<BR>
I need to somehow put a lot of that here! (huge work, as Optical Character Rcognition (OCR) is often problematic for me, even though it was pretty good at work)<BR>
<BR>




<HR><A id="Data, Methodolgy, Applications, Software"> </a>
<h2>Data, Methodolgy, Applications, Software</h2>

"$d_Fischer" =  "http://www.BillHowell.ca/economics, markets/Fischer, David 1996 The Great Wave/"
for the internet, but of course it is whatever directory path that you have downloaded the files of this webPage to on own system, if you do download.



<HR><A id="Data sources"> </a>
<h3>Data sources</h3>
<BR>
Primary : <BR>
	David Fischer 1996 "The Great Wave, Price revolutions and the rhythm of history"  New York, Oxford University Press, ISBN 0-19-505377-X<BR>
<BR>
I have NOT [purchased, read] : <BR>
	Wilhelm Abel 1935 "Agrarkrisen und Agrarkonjunktur: Eine Geschichte der Land und Ernährgswirtchaft Mitteleuropas seit dem hohen Mittelalter" Hamburg & Berlin 1966 appendix https://de.wikipedia.org/wiki/Wilhelm_Abel<BR>
<BR>
<BR>
I have NOT yet used : <BR>
	https://historicalstatistics.org/		<BR>
		many links to other data sources<BR>
	https://thehistoryofengland.co.uk/resource/medieval-prices-and-wages/<BR>
<BR>
Modern themes that illustrate this best : 
<UL>
	<LI>Stephen Puetz - Universal Wave Series (kind of like the system of 20+ Mayan calendars)
	<LI>Ralph Nelson Elliott
	<LI>Robert Prechter
</ul>
<BR>


<HR><A id="Methodolgy"> </a>
<h3>Methodolgy</h3>
<BR>

<A id="Photograph paper graphs"> </a>
<h4>Photograph paper graphs : </h4>

Collect digital images of graphs in the book
<UL>
	<LI>cellphone photos - using drill press in my tool shop to [raise, lower] cellphone
	<LI>clamps - to reduce curving of book pages when photo-graphing
	<LI>heavy-duty lamps for lighting - unfortunately not my "natural light" fluorescent, so pages are yellow-tan
	<LI>USB transfer of photos from cellphone to computer
	<LI>photo-files renamed using Figure [numbers, titles] adapted from the book
</ul>


<A id="GNU Image Manipulation Program (gimp) - Image preparation for each curve of each graph"> </a>
<h4>GNU Image Manipulation Program (gimp) - Image preparation for each curve of each graph : </h4>
GNU Image Manipulation Program (gimp) isn't the easiest, most intuitive software, but it's the kind of initiation-by-fire that later leads to you being happy that you didn't stick with lesser [image, video] software.  I cannot provide a users' guide here, but I will make [random, scattered] comments on the process.  Because gimp [wording, processes, menus] can be a bit "strange" it may help to have this as a guide (or not).  Of course, you may be using an entirely different image processing sare, in which case my instructions are useless for YOU.  But these instructions are actually for me, as there is no way I will remember which [software, tools, techniques, how-to] even six months later unless I use applications often (not usually the case).<BR>
<BR>


<A id="gimp - [rotate, crop] graphs from each cell-photo"> </a>
<h4>gimp - [rotate, crop] graphs from each cell-photo</h4>
 
<UL>	
	<LI>From [nemo, dolphin, other] file manager, open one of the cellphone photos in gimp.
	<LI>gimp menu -> View -> Zoom -> Fit image in window
		<UL>
			<LI>This allows you to see the full image, so you can easily select the graph in the next step.
		</ul>
	<LI>gimp menu -> Tools -> Selection tools -> rectangle select 
		<UL>
			<LI>left-click-hold-down at one of the "outer corners" of the graph, drag the mouse to the opposite corner
			<LI>release the mouse button, then keyboard "Control-C" to copy the rectangular-selected graph
		</ul>
	<LI>gimp menu -> File -> Create -> From clipboard
		<UL>
			<LI>a new image window will pop up
			<LI>if the new image is NOT good, just close the window without saving, and go back to select the region again
		</ul>
	<LI>gimp menu -> File -> Save 
		<UL>
			<LI>if the new image is good, save the window to the filepath of your choice (normally your working directory for this project).  IMPORTANT : the save defaults to gimp format (.xcf), and leave it that way so that you can use gimp to modify the image.  Once finished - you can export to [jpg, png, other] formats.
			<LI>Make copies of the cropped graph image for each curve that you will extract (assuming that the curves [cross over, include [dotted, dashed, otherwise] discontinuous curves, approach closely]).
		</ul>
</ul>


<A id="gimp - remove [adjacent, cross-over] curves for each curve-file : "> </a>
<h4>gimp - remove [adjacent, cross-over] curves for each curve-file : </h4>
 
This step ensures that in the next section, the digitizelt graph curve digitizer, will capture most, if not all, of the curve automatically with one attempt, which can save you a lot of time and frustration. 
<UL>
	<LI>From your file manager, open the curve-graph-file in gimp
	<LI>gimp menu -> View -> Zoom :  Zoom in on a section of the curve at a scale that is easy to work with.
	<LI>Select the color of the curve: 
		<UL>
			<LI>gimp menu -> Tools -> Colour picker 
			<LI>Place color picker over a segment of the curve, left-click.  This selects the current color of the curve. You can, of course, select a different color, but you must be sure to re-color the entire curve!
			<LI>My advice - ALWAYS write down the RGB values of the colors!!  This is important for possible color operations later.  Furthermore,p sometimes uses 0-255, and sometimes 0.0-1.0 (easy to convert as simple proportions). Oops - now I forget how I found the RGB values (later, some day, I will add it here...). Hint?? :
				<UL>
					<LI>gimp menu -> Windows -> Dockable Windows -> Tool Options 
					<BR>	<TABLE border=1>
							<TR><TD>background color :</td><TD colspan="2">Scanned curve</td><TD colspan="2">Price equilibria</td></tr>
							<TR><TD>Red		</td><TD>0.808	</td><TD>206	</td><TD>0.752	</td><TD>192 </td></tr>
							<TR><TD>Green	</td><TD>0.627	</td><TD>160	</td><TD>0.922	</td><TD>235 </td></tr>
							<TR><TD>Blue	</td><TD>0.486	</td><TD>124	</td><TD>0.215	</td><TD> 55 </td></tr>
							</table>
				</ul>
		</ul>
	<LI>gimp menu -> Tools -> Selection tools -> Free select (lasso icon?)
		<UL>
			<LI>I usually prefer this with the eraser, rather than the eraser alone, for removing other [curves, text] from the curve of interest.  The eraser jumps around to much, and is too crude to be efficient in most cases, but sometimes it is the best to use.
			<LI>Left-Click at a "clean end" of a segment of the curve of interest, but just at the side of the curve so as not to erase it.
			<LI>Release the mouse button and go to the other "clean end" of the curve of interest (or way off it to clear an area), and click again.
			<LI>Staying on the same side of the curve, click a perimeter that it sufficient to clear that [part, crossover] of the curve.
			<LI>Finish the perimeter clicking at the initial starting point.
		</ul>
	<LI>Erase everything within the perimeter by fat-pencilling over it with the right color. 
		<UL> 
			<LI>gimp menu -> Tools -> Colour picker
				<UL>
					<LI>click on a clean area of your graph that has the same background color that surrounds your curve.
				</ul>
			<LI>gimp menu -> Tools -> Paint tools -> Pencil
			<LI>I set up my mouse wheel so that "Alt-key + mouse wheel up-down" either [increases, decreases] the pencil's width.
				<UL>
					<LI>gimp menu -> Edit -> Preferences -> Input devices -> Input controllers :
						<UL>
							<LI>"Main Mouse wheel" should be in the "Active controllers" right-hand side.  If not select "Mouse wheel" in the "Available controllers" list and move (right arrrow in between the lists) to "Active".
						</ul>
					<LI>double-left-click on "Main mouse wheel" brings up "Configure Input Controller " window
						<UL>
							<LI>General : shows "Main mouse wheel"
							<LI>Select "Scroll up   (Alt)", click "Edit" button : I set this to "tools-valu2-increase"
							<LI>Select "Scroll down (Alt)", click "Edit" button : I set this to "tools-valu2-decrease"
						</ul>
					<LI>set a fat width to erase a wide swath.
				</ul>
			<LI>I use a big eraser, but I'm probably doing this wrong. I just want to bucket-fill with a color of choice, but I've [forgotten, lost instructions] of what to do as it doesn't work simply. 
				<UL>
					<LI>The eraser's action is limited to your free select area, which is exactly what you want, as a mistaken motion won't erase your curve or other important areas of the image.
				</ul>
		</ul>
	<LI>Repeat "Erase everything within the perimeter" all along the curve of interest
		<UL>
			<LI>This must "free" the selected curve from all potential interference.
		</ul>
</ul>


<A id="gimp :  curve, make [continuous, mono-color]"> </a>
<h4>gimp :  curve, make [continuous, mono-color]</h4>
 

<UL>
	<LI>As in the previous section, select the color of the curve: 
		<UL>
			<LI>gimp menu -> Tools -> Colour picker 
			<LI>Place color picker over a segment of the curve, left-click.  This selects the current color of the curve. You can, of course, select a different color, but you must be sure to re-color the entire curve!
			<LI>My advice - ALWAYS write down the RGB values of the colors!!  This is important for possible color operations later.  Furthermore,p sometimes uses 0-255, and sometimes 0.0-1.0 (easy to convert as simple proportions). Oops - now I forget how I found the RGB values (later, some day, I will add it here...). Hint?? :
				<UL>
					<LI>gimp menu -> Windows -> Dockable Windows -> Tool Options 
					<BR>	<TABLE border=1>
							<TR><TD>background color :</td><TD colspan="2">Scanned curve</td><TD colspan="2">Price equilibria</td></tr>
							<TR><TD>Red		</td><TD>0.808	</td><TD>206	</td><TD>0.752	</td><TD>192 </td></tr>
							<TR><TD>Green	</td><TD>0.627	</td><TD>160	</td><TD>0.922	</td><TD>235 </td></tr>
							<TR><TD>Blue	</td><TD>0.486	</td><TD>124	</td><TD>0.215	</td><TD> 55 </td></tr>
							</table>
				</ul>
		</ul>
	<LI>Mark the path of the curve (at least the parts that need to be filled in with the right color) :
		<UL>
			<LI>gimp menu -> Tools -> Paths
			<LI>left-click at one end of the curve, and click at each "turning-point"
			<LI>gimp's Bezier curves allow you to follow curved sections (see instructions)
			<LI>once you have accurately set the path : 
				<UL>
					<LI>gimp menu -> Edit -> Stroke selection ->  
					<LI>set [stroke line, solid color, anti-aliasing, linewidth >=3 to make it easy for digitizelt to follow the curve]
					<LI>left-click "Stroke" button
				</ul>
		</ul>
	<LI>Save the file in native gimp .xcf format (important for changes later to avoid re-doing work)
	<LI>Export the file in [jpg, png, whatever] format :
		<UL>
			<LI>gimp menu -> File -> Export As ->  
			<LI>Bottom right corner "All export images" drop-down list : select "jpg"
			<LI>set the proper [directory, filename], and change the file extension to match  te format that you want!
			<LI>Click "Export"
			<LI>Adjust quality of output image if desired
			<LI>Click save
		</ul>
</ul>
<BR>


<A id="Digitize graph curves using https://www.digitizeit.xyz/ software"> </a>
<h4>Digitize graph curves using https://www.digitizeit.xyz/ software</h4>

<UL>
	<LI>I selected this software from a huge selection, almost randomly
		<UL>
			<LI>minimum requirements - [try it before you pay, modest cost (~50 $US), command-line use 
			<LI>(30Jan2021  I haven't check the extent to which this can be run directly from scripts, as I am still learning the basics of software.)
			<LI>quick trials - [simple, fast to get going, effective]
			<LI>I am very happy with it
		</ul>
	<LI>In the recent past (my Puetz mini-project), I think I used https://automeris.io/WebPlotDigitizer/ 
		<UL>
			<LI>I couldn't [remember, find it] even though I may have paid for it.  This is typical for me, losing [track, memories] only a month or two after using it.  I found it soon after buying digitizelt, stuck with digitizelt.
		</ul>
	<LI>Digitizelt instructions are simple, so I won't comment.
	<LI>Key image preparations!! :
		<UL>
			<LI>Make separate copies of each graph for each curve that you will extract  IF the curves cross over or come into close proximity.
		</ul>
	<LI>NOTE :  I have not looked into scripting of digitizelt.
	<BR>
	<LI>To launch : 
	<LI>$ java -jar "$d_SysMaint""digitizelt graph from image/DigitizeIt.jar"  
		<UL>
			<LI>Note : "$d_SysMaint""digitizelt graph from image/" is the directory where I save the digitizelt system.
		</ul>
	<LI>Digitize a curve : 
		<UL>
			<LI>Digitizelt Menu -> Axes -> [X, Y] axes [minimum, maximum] - simply click on the right spots on the graph axes
			<LI>Digitizelt Menu -> Axes -> [X=linear, Y=logarithmic] 
			<LI>Digitizelt Menu -> Auto -> Options -> Automatic digitizing window
				<UL>
					<LI>Max. RGB differeence within the same object (marker or line): I set to 40-80% (possibly lower end with good-quality curves?)
					<LI>Symbol matching in % :  I haven't used symbols yet, so leave it be
					<LI>Max. number of digitized points on line : I leave at 10000
				</ul>
			<LI>Digitizelt Menu -> Auto -> Find Line
				<UL>
					<LI>If the curve is of good quality and easy for digitizelt to follow, it's possible that the entire curve is automatically digitized.
					<LI>Sometimes you may have to save data from each segment of the curve as it is digitized.  If you export the data for each segment to a file, then if something goes wrong you won't lose the segments that you have already done : 
						<UL>
							<LI>Digitizelt Menu -> File -> Export ASCII -> [directory, filename]
							<LI>Once all segments have been save to file, combine and sort the data, example : 
								<UL>
									<LI>$ d_gimped="$d_PROJECTS""References/Fischer, David 1996 The Great Wave/gimped/"
									<LI>$ echo  "year,Fig 0.01 consumables"  >"$d_gimped""Fig 0.01 Price of consumables in England 1201-1993.dat"  
									<LI>$ cat  "$d_gimped""Fig 0.01.dat"  "$d_gimped""Fig 0.01 2.dat"  "$d_gimped""Fig 0.01 3.dat"  "$d_gimped""Fig 0.01 4.dat"  "$d_gimped""Fig 0.01 5.dat"  |  sort  >>"$d_gimped""Fig 0.01 Price of consumables in England 1201-1993.dat"  
									<LI>This data is now ready (comma-delimited) for gnuplot. 
								</ul>
						</ul>
				</ul>
		</ul>
	<LI>Digitizelt Menu -> File -> Export ASCII -> [directory, filename] - the data to a file, if this has not already been done.
	<LI> sort the data, especially as curve segments will disorder the data (not essential, but easier to read).  Use an invert sort when dates are BC (negative years)! (use the spreadsheet to sort if data crossed [BC, AD])
</ul>	
<BR>


<A id="IMPORTANT issues with digitization of curves"> </a>
<h4>IMPORTANT issues with digitization of curves</h4>

Digitization of curves is not without its issues, and this is NOT limited to https://www.digitizeit.xyz/ software, although different software may be [more, less] robust with respct to these problems.  Digitizelt probably has settings that can be adjusted for particular datasets, but I have not looked for that in my first use of the software (later, maybe much later...).  In a nutshell, the digitization must be [robust, accurate] on the whole, and compromises are required in the software.  "Peak shaving" (high and low) occurs, and high frequencies tend to be smoothed out, and these can be critical to some applications.<BR>
<BR>
<UL>
	<LI>The axis are typically at angles, with curvature from the cellphone photo.  This causes minor  distortions in the data. 
	<LI>It is a very simple process to go back to the dataset and manually correct for "peak shaving", as only a small fraction of the data is affected (usually). But I haven't done this. 
	<LI>Retention of high frequencies can be much more manual work, depending on the curve. 
	<LI>I end up with a ridiculous number of "data poinst", as each line segment in the original data can have humndreds of "false data" points! This can be removed, with work, and will ultimately be necessary if the spreadheet grows too large (it's already inconvenient).  "Jitter" of digitized points on a straight line is observable.
	<LI>Fischer's Appendix H "Nominal prices and silver equivalents" provides a very useful discussion of [how, why] historical studies sometimes correct for monetary fluctuations, especially currency debasements.  Data is sometimes reported on a "nominal" basis, and there is likely an issue with the [stability, inflation] of the metric (eg dollar in modern times, grains of silver or gold as mining [reserves, costs, availability] change).
	<LI>Never forget that the preparation of the data by the original authors also will involve simplifications, and also involves data [filtering, smoothing] techniques. Regional variances within a country a also important, as pointed out by Fischer.  I have NOT checked the sources for these issues.
</ul>
I have NOT done this simple work!  Yes- laziness creeps in, but it's just an initial stage of the project, and this can be done later (yeah right, but will I do it?)<BR>
<BR>


<A id="Warehousing graphical data"> </a>
<h4>Warehousing graphical data</h4>

"$d_webRawe""economics, markets/Fischer, David 1996 The Great Wave/1_Fischer 1200-2020.ods" : <BR>
-add sheet for each graph-curve<BR>
<BR>
I typically use a LibreOffice Calc spreadsheets to [collect, rearrange, simple transform] data.  For this project : <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/1_Fischer 1200-2020.ods">1_Fischer 1200-2020.ods</a><BR>
<BR>
For gnuplot, it is most convenient (albeit not necessary) use comma-delimited (csv-like) text data files.  Just save the spreadsheet in that format, so now you have a data warehouse with the [convenience, power] of spreadsheets, and simple output usable by gnuplot.<BR>
<BR>


<A id="Set timePoints for transitions between price [revolutions, equilibria]"> </a>
<h4>Set timePoints for transitions between price [revolutions, equilibria]</h4>

The [first, key] step is to select the timePoints for transitions between price [revolutions, equilibria]. The endPoints of price revolutions was taken at the point wherrices stabilised following a price revolution "crash" and tended to drift lower slowly.  Price equilibria endPoints were taken to be where significant crease first occurred,  and the subsequent price revolution begins, somewhat slowly.<BR>
<BR>
<BR>  These timePoints are somewhat "cloudy", and while I kept "close" to dates as usually specified in Fischer's book, I didn't take these as exact.  Perhaps the actual price data are a better reference.  I used the "consumables prices" as a basis for other curves (eg grains) for the medieval to modern periods, but it's br to treat each curvce separately.<BR>
<BR>
This is susceptible to serious bias in selecting the [start, end] dates for each segment. See the spreadsheet <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/1_Fischer 1200-2020.ods">1_Fischer 1200-2020.ods</a>.<BR>
<BR>
For example, I initially selected the following end-points fREngland's consumable prices :  <BR>
year	1206.58	1373.55	1501.78	1661.06	1754.20	1803.78	1937.77	1999.74<BR>
<BR>
The year ~1940 was taken as the [start, end] point for my <A HREF="!!linkError!!">1872-2020 S&P 500 stock index [start, end] point, so I use it here as well.<BR>
<BR>
We don't know [how, when] the "modern price revolution" will end, so I've merely used the last data-point.<BR>
<BR>
<BR>


<A id="Do single-linear-regressions of price [revolution, equilibrium] segments"> </a>
<h4>Do single-linear-regressions of price [revolution, equilibrium] segments</h4>

This is easy with the spreadsheet - one column of regression results I use 10 year intervals per segment, but you only really need the [start, end] dates [-,+] 20 years.  The extra 20 years extends the segments at both ends for visual clarity. For an example, see the spreadsheet <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/1_Fischer 1200-2020.ods">1_Fischer 1200-2020.ods</a>, sheet "Fig 0.01 SLRsegments".<BR>
<BR>
Save the "SLRsegments" to a data file that canused by GNUplot.  Example :  <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/Fig 0.01 line segments for GNUplots.dat">Fig 0.01 line segments for GNUplots.dat</a>.  Notice that olumn titles can use free-format text, except for the comma, which separates columns.<BR>
<BR>
It would greatly reduce annoyance if this entire step was automated.<BR>
<BR>
I used my simple Qnial programs to do the linear regression : 
<UL>
	<LI>Save data of <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/1_Fischer 1200-2020.ods">1_Fischer 1200-2020.ods</a> to data file , example <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/Fig 0.01 linear regression raw data.dat">Fig 0.01 linear regression raw data.dat</a>
	<LI>For QNial the data in rows must be separated by tabs, and the column titles must : be separated with a quote ("), not include [spaces, punctuation, strange characters] (use underline (_) as a substitute).
	<LI>For each curve, <A HREF="!!linkError!!Fischer linear regressions.ndf">Fischer linear regressions.ndf</a> - a special operator (proceedure) is created to select a segments's dataFile, handles [data, results] file [input, output], and calls <A HREF="[#=; backtrack ;=#]Qnial/MY_NDFS/fit_linearRegress.ndf">fit_linearRegress.ndf</a>
	<LI>Copy the [constant,slope] for each segment to "1_Fischer 1200-2020.ods", sheet example "Fig 0.01 SLRsegments". 
	<LI>"Fischer linear regressions.ndf" saves the estimated ression results, Yest, results to a text file, which is then copy-pasted to "1_Fischer 1200-2020.ods"
	<LI>In turn, "1_Fischer 1200-2020.ods" data is once again exported in csv format (including the Yest results) that GNUplot can use (I could also use tab or other delimited, not just commas).
	<LI>Note that the use of the spreadsheet is not essential and does require "double-handling" of the results, but it does provide a nice "data warehouse", amking it far easier to follow results especially years after doing this work (or if someone else wants to use it).
</ul>
<BR>


<A id="GNUplot - create graph of nominal data"> </a>
<h4>GNUplot - create graph of nominal data</h4>

30Jan2021 - I'm too lazy to provide details at this time, but the good thing is that I used gnuplot scripting, so this part is perfectly well described by scripts (in this web-directory) and the datasets.  Examples are :
<UL>
	<LI>text data file : <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/Fig 0.01 Price of consumables in England 1201-1993.dat">Fig 0.01 Price of consumables in England 1201-1993.dat</a>
	<LI>gnuplot script : <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/Fig 0.01 Price of consumables in England 1201-1993.plt">Fig 0.01 Price of consumables in England 1201-1993.plt</a>
	<LI>graph output	 : <A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/Fig 0.01 Price of consumables in England 1201-1993.png">Fig 0.01 Price of consumables in England 1201-1993.png</a>
</ul>
For this project I did not write bash scripts that create the gnuplot scripts, but I've done that in the past.  It's a life-saver for repetitive work.
<BR>


<A id="[calculate, plot] detrended results for each time segment"> </a>
<h4>[calculate, plot] detrended results for each time segment</h4>

During the step "Do singular linear regressions for each time segment" described above, the QNial program  generates Yest values for each digitized point.  These values are copy-pasted to the "data warehouse" spreadsheet, where a simple formula calculates detrended values.<BR>
<BR>
Note that I calculate 10* the detrended log values, which was used a previous project for the S&P 500 stock market index.  From that community I have adopted a [fractal, Fibonacci] scaling of the outputs on a [-5.7] interval.  Fibonacci levels (and related Elliot waves) are very widely used in technical analysis for trading, and I apply it to the historical series as well.<BR>
<BR>
<BR>
The main detrended plots are defined in GNUplot files :
<UL>
	<LI><A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/Fig 0.01 Price of consumables in England 1201-1993 detrended.plt">Fig 0.01 Price of consumables in England 1201-1993 detrended.plt</a> - This covers the medieval to mdern era, and is used to colect curves for different data.  The restricted t-frame provides a more accurate view of that period.
	<LI><A HREF="[#=; backtrack ;=#]economics, markets/Fischer, David 1996 The Great Wave/1850 BC to 2020 AD prices detrended.plt">1850 BC to 2020 AD prices detrended.plt</a> - Obviously this covers a variety of regions, time-frames.  What I really need is data going 7,500 years (~3 cycles of 2,400 years (Halstatt cycle) corsponding to a 2006 project on the rise and fall of civilisations <A HREF="[#=; backtrack ;=#]Civilisations and sun/_Civilisations and the sun.html">_Civilisations and the sun</a>, and if I find [time to do it, data] this would be nice.  
</ul>
<BR>



<A id="Simple template for working on a dataset"> </a>
<h4>Simple template for working on a dataset</h4>

08********08<BR>
ddmmmyyyy Template for data analysis steps<BR>
<BR>
1.	copy section to new date near start of this file <BR>
2.	copy-replace "Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC" <BR>
......with the new photo-image-curve name<BR>
3.	careful to update information as you work<BR>
NOTE :  
d_gimp="$d_PROJECTS""References/Fischer, David 1996 The Great Wave/gimped/"
d_Fischer="$d_webRawe""economics, markets/Fischer, David 1996 The Great Wave/"
However, most applications REQUIRE the full path!!
<BR>
<BR>
+-----+<BR>
gimp : crop graph, curve [separate out, make [continuous, mono-color]]<BR>
/media/bill/Dell2/PROJECTS/References/Fischer, David 1996 The Great Wave/gimped/Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC raw image.xcf<BR>
<BR>
/media/bill/Dell2/PROJECTS/References/Fischer, David 1996 The Great Wave/gimped/Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC raw image.jpg<BR>
<BR>
+-----+<BR>
digitizelt : digitize graph curves using https://www.digitizeit.xyz/ software<BR>
$ java -jar "$d_SysMaint""digitizelt graph from image/DigitizeIt.jar"  <BR>
<BR>
/media/bill/Dell2/PROJECTS/References/Fischer, David 1996 The Great Wave/gimped/Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC raw data.dat  <BR>
<BR>
Use an invert sort as number are negative! (use the spreadsheet if data crossed [BC, AD])<BR>
$ sort  --reverse "$d_gimp"Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC raw data.dat"  >"$d_Fischer""Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC sorted.dat"<BR>
<BR>
Add column titles : must be QNial phrases - replace [space, punctuation]s with underscore `_<BR>
+-----+<BR>
QNial : process digitized data<BR>
<BR>
add_logCol link  d_Fischer  fNameBase  ' sorted.dat'<BR>
paste to : graph-curve sheet "Fig 0.02"<BR>
column titles : must be QNial phrases - replace [space, punctuation]s with underscore `_<BR>
column separator : tab<BR>
<BR>
copy log formula down "log_price column<BR>
copy : titled-data  ["i_dat	"year	"Fig_5_01_prices	"log_price]<BR>
to	  : "$d_Fischer""Fig 5.02 ancient Greece [barley, olive oil] 450-150 BC, QNial format.dat"<BR><BR>
<BR>

+-----+<BR>
Set timePoints for transitions between price [revolutions, equilibria], Linear regression of price [revolution, equilibrium] segments<BR>
<BR>
onset timePoints :<BR>
equilibrium	revolution	equilibrium	last_data<BR>
-1796.061	-1739.991	-1684.230	-1624.393<BR>
<BR>
+-----+<BR>
Do singular linear regressions for each time segment<BR>
<BR>
Use 
"$d_Fischer""Fig 5.01 linear regression raw data.dat"  <BR>
<BR>
"$d_Fischer""Fig 5.01 prices in ancient Babylon 1850-1600 BC.dat"  <BR>
<BR>
+-----+<BR>
GNUplot - create graph of normal data<BR>
<BR>
"$d_Fischer""Fig 5.01 prices in ancient Babylon 1850-1600 BC.plt"<BR>
<BR>
+-----+<BR>
[calculate, plot] detrended results for all curves analysed to date<BR>
<BR>
"$d_Fischer""1850 BC to 2020 AD prices detrended.dat"<BR>
<BR>
<BR>


<A id="Software tools - general comments"> </a>
<h4>Software tools - general comments</h4>

Here are the key [image, graph] tools that come to mind :<BR>
	<table border=1>
	<TR><TD><A HREF="https://www.digitizeit.xyz/">https://www.digitizeit.xyz/</a>	
		</td><TD>to extract data from digital images of graphs</td></tr>
	<TR><TD><A HREF="https://www.gimp.org">https://www.gimp.org</a>		
		</td><TD>GNU Image Manipulation Program - [free, very powerful, etc, etc]<BR>
					I have used gimp "scripts" - mostly for video aimation production</td></tr>
	<TR><TD><A HREF="http://www.gnuplot.info/">http://www.gnuplot.info/</a>
		</td><TD>GNU plot - [free, very powerful, etc, etc]</td></tr>
	</table>
<BR>
My basic toolset for much-of-what-I-do : 
<UL>
	<LI>[Unix, bash] cart [environment, command, script]s 
		<UL>
			<LI>example commands : [find, grep, sed]
			<LI>piping, I/O redirection, etc, etc.
		</ul>
	<LI>[geany, kwrite] text editors for programmers, especially : 
		<UL>
			<LI>regular expression [search, replace] -  How could I have been so slow to [learn, adapt]?  Years of wasted effort and stupidity!!!  I should have listened to friends.
			<LI>"vertical" editing across lines.  I will NOT work without that anymore!
		</ul>
	<LI>Queen's University of [Kingston, Ontario, Canada], Nested Interactive Array Language (Q'Nial)
	<LI>LibreOffice Calc spreadsheet program -  This is equivalent to Microsoft Excel, and is fairly compatible other than scripting.  Word is a very high quality application, no doubt, but it costs far too much (forget their bozo home editions) and it belongs to one of the evil empires (another evil empire is Apple, but there are plenty more).
	<LI>gnuplot -  I realize after far too many decades that I have been a complete moron for using the graphical capabilities of spreadsheet programs.  Far too often the use of spreadsheets themselves is also moronic, for example when text files can be far more [open, powerful, effective], but spreadsheets are  great data warehouse, allow far [faster, easier, ad-hoc] transformations of data than any other means, and they do the best job of making data visible and transformable (up to some data [dimensionality, size] when visual becomes clumsy or problematic).
</ul>

My requirements for software : 
<UL>
	<LI>Must be able to run entire program from a bash script or any other programming language of your choice, like Q'Nial that I also like, or as other examples C, forth, LISP, python, pearl, java, whatever...  
	<LI>Cheap - like free...
	<LI>Graphical User Interfaces (GUI) are great when you are just starting to learn software and it's capabilities.  After that, and apart from many specialised tasks where GUI shines (like gimp), GUIs are just a pain in the ass and get in the way of doing real work [en mass, fast, of quality, reliably, that are [transferable, general]].  I imagine that many people require GUIs because they suck at scripts (i.e. myself not so long ago), or when they use the software only rarely, with plenty of time between uses to forget everything.  In that later case, scripts are a vastly superior approach, except of course when people are not familiar with scripting.
</ul>
<BR>




??Jan2020 initial posting<BR>
[#!: full-line executeEmbeds,  phraseValueList = (("fout  fout)("backtrack  backtrack))
[#!: path_executeEmbedsInsertIn_fHand  (link  d_webWork  'fin Footer.html')  phraseValueList  ;