-
Notifications
You must be signed in to change notification settings - Fork 6
Expand file tree
/
Copy pathNotes.creon
More file actions
executable file
·1585 lines (1206 loc) · 58.1 KB
/
Notes.creon
File metadata and controls
executable file
·1585 lines (1206 loc) · 58.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
should this look at annualized returns, or scale returns by dt
should plot rank insted of value?
should we plot (manhattan) distance between parameter values?
what should we eliminate durin preprocessing:
1) guys who are <=0 profit at t=0.1
2) what else?
Color by value at t3.
animate slider
smart renorm (preserving origin, like global norm)
X t2 = t1 + dt <= tf
X must redraw whenever buttons or sliders change
Direct selection of nets.
Add "selection only" display
display profit history (parallel coords)
add new kind of window subclass for showing selected net leraning parameter scatterplot
-----
Jacques says:
class c2 is subclass of c1
class c1 has method c1::ma()
c2 does not have its own ma()
c1 and c2 each have their own separate methods mb()
c1::ma() calls mb()
i1 = new c1
i2 = new c2
i1.ma() will call c1::ma() which will call mb() which is c1::mb()
i2.ma() will call c1::ma() which will call mb() which is c2::mb()
Well, that works, but only if mb is declared virtual in c1.
-----
Install fltk 1.1.5
Keyboard access to slider values
control panel per window (or control panel settings per window)
t1, t2, displacement?
---
rgb color exponent slider
Precompliled headers for blitz and fltk (and gmm++)
tried it - only saves a a couple of seconds on compile
link gmm++ to blas and lapack
ratios of densities
matrix invariants, or other anisotropy measures
eigenvectors?
least squares grad and hessian
relative coords.
natural neighbors densities
control panels consolodated into one, with one tab per plot
fix normalization, etc. For pairs, xforms, etc.
normalization (scaling of data) seperate from display offset, except perhaps for 3sigma
e.g. do not shift data zero - just scale data, have separate option for display -1:1 or 0:1
add sixsigma?
histograms for marginals
make skelaton for other programs, e.g. ktest
multiple selections
separate colors for selected, unselected, blendcolor
panel buttons to duplicate keys to reset xform, invert selection, deselect, etc..
trackball 3d manip
binned kernel densities
nD trackball
running entropy / MI estimates
run entropy code
principal componenents
independent components
---
use texture (1-d?) coord instead of color to render color of points. Generate texture based on selection region, thus avoiding testing all points to see if they are in the region. Question: how to handle individual colors ramps and alphas on a per-plot basis: Answer, use a separate texture for each plot, but use the same texture coords. Use vertex (and texture) arrays to speed rendering.
---
denb_16, vs lam2_16, rank, x+y v x-y
select rhs vs lhs
10:00:09 05 May 05
Fast selection idea. Use mouse x and mouse y to create a new
texture (or texture matrix) that is effetively a blue rectangle
(corners based on mouse coords) inside a red square. Use each point's
x,y coords in active window (where selection is being made) as that
points texture coords in ALL windows. This can generalized to create
textures (coresponding to soft selections) based on arbirary functions
of coordinates and other (e.g. slider) variables. E.g. a texture coord
can represent energy, or distance from some n-d point, or how many
times a region has been brushed by an airbrush, etc....
18:57:25 06 Sep 05
Functionality to add (see also project plan):
***tuning
- remember normalization & ranking
- don't repeatedly rerank and/or renormalize the same var
- fast reranking after "delete selected points"?
- don't require multiple copies of vertices
- interleaved vs. not
- histogram speedups
***misc
- tile windows correctly for multiple screens
- dynamically couple to running sims on another machine(!)
- comments in files ("#")
- mapped files?
***global controls to non-tabbed panel
- npoints DONE
- start, end, stride for npoints
- random points (repsecting start & end)
- reset all views
- clear selection DONE
- complement selection DONE
- dont display deselected DONE
- add to selection BUGGY
- subtract from selection
- brush selection
- move selection
- new selection (as opposed to brush or move, works for add & subtract)
- delete selected points DONE
- write data to file DONE !
- global versions of local controls (size, nbins, etc)
- show count (and %) of selected/visible/total points
***per-view controls, in tab
- reset view
- selection color (or should this be on global panel?)
- better axis labels
- labels smaller
- labels display axis transformations (x+y, polar, normalizations, ranking...)
- tickmarks on axes (per axis?)
- polar grids?
- logarithmic axes
- via normalzation option
- show in labels
- atanh (logistic) normalization
- histogram enhancements
- speed them up when changing bin counts
- should be very fast for equi-depth case, when data is sorted: O(nbins).
- histogram bin counts independent for x and y
- default bin counts using Kevin Knuth's algorithm
- equi-depth option
- extend KK's algorithm for equi-depth case
- offsets for histogram
- better coloring for histograms
- better alphs, or edges on lines
- colored same as selection color
- selection via histograms
- boxplots
- density estimates
- binned kdes?
- point sprites (same thing?)
- multiple axes
- ND rotation
- time series
- vector variables
- rework normalization
- separate scaling, centering, windowing
- separate axes and windowing ?
***plot window enhancements
- min, max, mean, median, variance, stddev, quartiles, etc, (like R.summary())
- for each axis
- for each selection on each axis
- for each histogram bin on each axis
- boxplots
- multiple selections
- selection color chooser (global & local)
- fix relationship between color, depth, seletion visibility
- for 2D, alpha=0, selected points should never be obscured
- separate 3rd axis=NONE to force 2D
- then we can use z to make selected points draw on top of non-selected points
- tie plot window keys to control panel keys (shortcuts?), where possible
- trackballs for 3d rotation
- scale w.r.t. mouse down as center
- move point to center ?
- show center glyph, like flashearth
- lock aspect ratio to 1:1
- even when window aspect ratio is not 1:1
- fancy texture hacking, as above
***additional kinds of vis
- parallel coordinates
- vectors
- glyphs
- color by scalar
- MDS
- laplacian eigenmaps, LLE, other dim. reduction....
- classification (SOM....)
- co-clustering, info bottleneck, real gaussianization
- new random choice for axes
- slider for axis
- choose axes by MI
- high-d rotations
- predfined views (45 degree rotations....)
- projection pursuit in multiple plots at once.
- linked scaling/translation on idenitcal axes in separate plots
.....
Priorities (?)
histogram rescale slders
data scales (done, almost)
resizable windows (fix) (done)
mouse buttons test for translate & rotate, fix
rotation of (2D & 3D) axis
multiple tickmarks
normalization NONE -> zero to center of screen, -1 to left, +1 to right
Thu Sep 22 2005
fix centers too, in normalization routines
fix no redraw in (some) other windows when selection changes
fix z normalization (z-scale?) - rotation seems screwed up (done)
add scale numbers
tickmarks......
add priority to selected when in 2d (use Z = z+1 for each new selection)
texture based selection "colors"
timeseries
randomize vars ?
histo scale sliders!
swapbuffers timing
11:04:38 30 Sep 05
Timeseries / spectra (UGRIZ, orderbook, etc) ideas.
- timeseries (including timeseries of spectra)
- timeseries are handled by utilizing row index of point as timestamp.
- offset value (slider) for an axis displaces its row index values i by a constant k, so we can plot x_i vs. x_{i+k} (or vs y_{i+k})...
- npoints slider will need min, max, and "offset" sliders. offset rigidly moves min and max.
- spectra (books, parallel coordinates)
- spectra are stored for each point in its row (like ugriz).
- choose vis type (parallel coords, scatterplots...)
- for parallel coord vis type, column index is used as (implicit?) axis
- need to be able to choose (limits) for row indices that will be used as a set of parallel coordinates
- is there an fltk mutiple chooser we can use for this? (yes - class Fl_Multi_Browser ?)
- for parallel coord vis, plot glLines instead of glPoints
- use column labels as axis tick labels
- how to handle selections?
17:19:37 01 Oct 05
Simplest "parellel coord" implementation - levl 0
- extract vertices as huge array, 3*npoints*nvars in size, x=all vav values, y=column index, z=row index (or 0).
- no good. back out.
Added new starting axes (nonrandom). Added "choose new axes" which changes all axes to new ones.
- this could be on a per plot basis, too.
lock axis
- global
- per plot / axis
Tue Oct 4 11:26:29 2005
controls should have local and global versions
add trackball controls
data ideas
chemistry
final.txt DR4
+ add xyz (done)
+ add colors (both k-corrected and raw)
+ include far colors (u-r, G-K, etc)
+ rename z->spectro_z, M-z->z', M_Z->Z
+ add extra vars from vagc
+ add extra vars from photo, spectra
+ add extra vars from aux VAGC files
+ add density estimators from knn
+ add moment of intertia eigenvalues & summaries...
+ binned KDEs and derivatives
CFD / etc from tim?
machine learning data repository
4D rotation from one 2D plot to another
- frame interp. ala Asimov
- use opengl w/4d vertices
- use 4D rotation matrix: (s = sin(theta), c = cos(theta))
c 0 s 0
0 c 0 s
-s 0 c 0
0 -s 0 c
- generalization: use mouse-x, mouse-y to control two rotation angles, t1 & t2, and use matrix:
with c1 = cos(t1), etc.
c1 0 s1 0
0 c2 0 s2
-s1 0 c1 0
0 -s2 0 c2
- both plots get same set of 4d vertices, one plot uses the above matrix, the other plot uses
the above matrix with upper & lower half swapped. Or, swap x1,x2 with x3,x4 in 4d vertex array.
-- need "90%" and "99%" normalization styles, which trim the top 1% and top 10% of outliers, repectively.
-- change "add to selction" to "or selection"
- add "and with selection" (or some such wording)
15:05:22 02 Oct 05
- "display deselected" should be local (optionally). How about "d" vs. "D".
09:02:56 05 Oct 05
- "The Dependence of Clustering on Galaxy Properties" http://xxx.lanl.gov/abs/astro-ph/0509873 uses: (in addition to M_r(0.1)) D_4000 (which is "better thatn color g-r), H_alpha, "stellar" Mass M_*, surface density d_*, concenntration index C (as a proxy for hubble type),
Thu Oct 6 17:40:13 2005
- need local versions of "display deselected".
- need size for selected.
- need to spped things up. Bogs down on > 1M points.
- e.g. (multi)-texture idea. No "color-array-from-selection()", but a 2D texture is painted using selection tools, and that 2D texture is applied to all points. Selections in multiple windows are handled with multi-textures.
- LIBRARIES WRONG ON LINUX10? I need to link with /usr/lib/libGL.so.1.0.7667 and *not* with /usr/lib/libGL.so -> ../../usr/X11R6/lib/libGL.so
(see ftp://download.nvidia.com/XFree86/Linux-x86/1.0-7676/README.txt)
11:20:26 09 Oct 05
* - the best way to render this data with openGL on linux and osx is probably to use vertex buffer objects (VBOs). That way, the vertex coordinates can be static (they hardly ever change), and the color (or texture) coordinates can be dynamic (if necessary). There would need to be N+!, probably. N static ones for the vertex data and one for the (dynamic) color or texture data.
DONE - using textures insetad of colors for selection.
- initial imp; one selection color only.
- then, one texture for "show deleselcted", another for "dont show deselected".
- then, handle multiple colors and "add to selection".
- then (maybe) handle "don't paint".
17:35:15 04 Feb 06
features to add:
get good data for SDSS, genetic (www) domains, statarb, china study, cfd (alan?), chemistry, ....
DONE multiple colors
color = nwindows...2*nwindows-1 as t coord with one additional intersection color at t=2*nwindows. Add colors valuess for multiple selections, clamp t at t>=2*nwindows.
line plots
vector variabls
parellel coords
subject-observer pattern for multiple views
can we use sprites and colors?
how to use pixel textures for hue maps/lightness maps.
better blending
more dynamic range on blending
slider for points_color (again);
selected points should render on top of non-selected points.
are there too many redraws? Why do plot windows swap one at a time?
scale for z axis needs to be sensible
global reset to initial state
* add trackball rotation for 3D
separate 3D and 2D rendering?
separate histogram scale sliders?
separate histogram nbins sliders?
Jeff's default histogram nbins?
how to handle multiple screens?
windows overlap on macs is still wrong.
is there a monospaced sans serif font?
selection stats are too small, and should be rendered with logicop, or above main points.
need global control panel
size
normalization
colors?
blend?
lock axes
.....
chooser to omit certain panels from global ops.
need per axis control to skip certain axes on "change axes"
npoints stride? random?
DONE skip columns that only have one value (cmd line arg to override)
bail if too many input errors in a row..
comments in input files?
fine control on sliders - how?
For demo at apple:
DONE fix initial startup color (why is it white?)
DONE initial startup normalization trim 10^-3
DONE set z scale to x&y scale
DONE fix "reset view"
DONE get and use 2 screen info on mac
DONE (I think) true multiple colors
profile and tune.
DONE eliminate columns with no information
global panel
DONE get slides, data, and DEMO! together
14:29:08 15 Mar 06
todo list from eeting with Paul
Creon
get svn up on trotsky
move repository to trotsky
get windows version checked in
get svn on paul's machine(s)
get windows version backported to mac & linux
global reset to initial state
new todo list
single slider (or two, hue and sat) for color selector
complete splom
row number as implicit column
"data set" (see below) as implicit column.
order plots by mutual information
table of values for selection (alternate view)
all values (max rows)
basic stats
remove don't paint
clear selection on new windows, always, or never
Paul
vp web site on trotsky
finish class file splitting
fix binary IO
add header processing
create/destroy plot windows
reload input file
create new window or reset to initial window configuration
add row / add column
menu bar design
compile with optimization
startup with no data & from icon
Paul & Creon
binaries to vp web site
read more data (on the fly, or from command line).
1) add another column
2) add more rows, with an implicit column for which data set.
missing values
10:58:51 01 Apr 06
Mon Apr 17 11:57:13 2006
=======
Mike's suggestions
screenshots for web, manual, and PASP/astro-ph
write a FAQ
what commands to run if vp doesn't work.
wiki
opengl extensions registry
linux library checking
send boost config file library to paul
dependencies
sample data
list of known platforms
=======
1. command line args to specify which variables to plot, initially
2. command line args for initial selection bounds
3. current position of mouse should be dynamically printed (like the selection statistics)
4. separate x and y histogram bin count controls
5. separate point size control for selected points
in 2D, this should be easy with glPointParameterfARB
could be different for each selection.
better with NV_Point_sprite
- Then we can have fast open circles, filled circles, squares, X's, +'s, letters, Gaussian humps....)
15:58:48 02 Apr 06
Ideas from simvis http://www.vrvis.at/simvis/
smooth brushing
(smooth) brushing in histograms
simple (greyscale) luminance composition for deselected points
ideas from ggobi/rggobi http://www.ggobi.org/
"replicates"
"edge display" / edge brushing (see movies of linked brushing on website)
"persistant" vs. "transient" brushing (i.e. "paint/add to selection")
Wed Apr 26 16:11:57 2006
New idea for spectra, lineplots, etc:
in the axis choice menu, allow for multiple selections. Select a contiguos group
to plot a spectrum or timeseries (instead of a point).
16:42:58 24 May 06
NOTE log(voronoi_volume) for DR4 final has Gaussian distribution. Why is voronoi volume distribution lognormal? at amy and all redshifts?
Z-axis option should be "selected" - so selected points are plotted above nonselected (turn on depth buffers)
need per plot "deselected color" control
need to use "selected" (float) as 3rd coordinate for 2d plots.
add "zoom to selected set" (zoom so selection bbox fills window)
add equi-depth histograms
add shifted histograms
Look at IRAS data
12:19:42 16 Jun 06
notes from meeting w/Paul
selector (with scrolling) for axes, instead of popup menu
mulitple selection in y axis selector selects (implicit) column index for x axes, and plots lines!
normalization either globally (across all y values) or per column (parallel coords style)
binary search on ordered data to find selected set, instaed of brute force range check
stack trace on erroc (see gnu libc backtrace() ) *built in on macos*
z = selected option
pointpramameters for size
catagorical data (via hash)
look into:
can you grow a blitz array (e.g. when reading file)
boost (or standard) arrays instead of blitz++ *not yet*
fttk scripting
email to paul re finance (talk with Adrian smith and Sandra Owen)
get GLSL book (especially pipline reference), openGL guide fifth edition, *done*
can use vertex program to pass vertex component arrays separately.
and to do n-D projection/rotation
ideas:
use select to move indices between selected and nonselected arrays. *done*
then we can render selected and deselected separately, to get selected "on top" in 2d plots.
can us opengl array index primitives. can stll use vbos(s)
can use color call, once per index array, instead of one texture coord per vertex.
using color, we can have colored point sprites using standard point sprites (no need for r coord)
can have multiple arrays, for multiple selections.
try point smoothing again. *done*
add outlier detection.
handle alternate field delimiters with --delimiter="," etc...
make a function convert_delimiters_to_whitespace(string)
but this doesnt allow us to handle missing values (1,2,3,,5,,,8,9)
for now, if there aren't enough variabes read on one line, then assume that line has missing values and discard it (with a counted log message).
Idea:
toggle through multiple selections, like invert selection, but next plot's selected set (only) is shown. deselected is also shown (?). Plots with no selected points get skipped.
We can visualize multiple time series (and spectra, etc) in multiple variables as follows: assume we have series
S_1 = {{x1_1, y1_1, z1_1}, {x2_1, y2_1, z2_1}, .... {xn_1, yn_1, zn_1}}
S_2 = {{x1_2, y1_2, z1_2}, {x2_2, y2_2, z2_2}, .... {xn_2, yn_2, zn_2}}
...
S_N = {{x1_N, y1_N, z1_N}, {x2_N, y2_N, z2_N}, .... {xn_N, yn_N, zn_N}}
create a file with the following format:
series timestep x y z
1 1 x1_1 y1_1 z1_1
1 2 x2_1 y2_1 z2_1
...
1 n xn_1 yn_1 zn_1
2 1 x1_2 y1_2 z1_2
2 2 x2_2 y2_2 z2_2
...
2 n xn_1 yn_1 zn_1
....
N 1 x1_N y1_N z1_N
N 2 x2_N y2_N z2_N
...
N n xn_N yn_N zn_N
note that the timeseries (spectra) do *not* all have to have the same number of timestpes (wavelengths), nor do they have to be equaly spaced.
Q1) is there a way to turn a more traditional spectral data fle inot this kind of file by "folding" on input in some clever, autofocus sort of way?
Q2) How do we (efficiently) optionally connect draw the individual timeseries as polylines?
Idea - add "data table" viewer for selected points. (Joe Mazzarella at ipac)
Better color selection:
Saturation is the second "contouring" control I've been waiting for.
Add saturation & hue (quantized) to main control panel tab. (& global tab)
Remove selection color popup. (& deselection color popup)
h,s,v,a sliders (with optionally quantized hue) for everything.
Better direct manipulation:
clicking on axis label in plot window pops up menu of axes
clicking and drag on tick mark drags is (& rescales plot)
click
Get Tycho 2 catalog (http://www.astro.ku.dk/~cf/CD/) and use it for demos. 500M download.
For fast selections, use binary search on sorted values on at least
one axis to esablish bounds for linear search. This will even work for
non-rectangular brushes.
Interface to weca (ML application) using plastic protocol.
Simple - legend for colors to density map.
Add pallette of hues to color chooser.
"Error bars" by associating additional variables (low priority, probably better ideas).
15:58:44 23 Oct 06
VBO strategy:
first, put vertex coordinates into VBO, leave indices in user memory.
put all nplots*npoints vertices into one VBO, thus we only need to worry about one binding.
only need to bind one buffer, to target GL_ARRAY_BUFFER, usage GL_STATIC_DRAW
use glBufferSubData() to put an individual plot's vertex coordinates into the VBO.
OK - this initial VBO strategy seems to be working, at higher
performance, too. Now, we need to solve the problem that changing
nplots (i.e. adding and removing rows or columns of plots) causes a
crash. This is because we need to release and then recreate the VBO
to be larger/smaller according to whether nplots increases or
decreases. This may also be an issue when npoints increases/decreaes,
for example by killing points or (soemday) appending points. Anyway,
we will need to rexecute the logic in initializeVBO when things like
this happen. (i.e. a new call to glBufferDataARB() with more/fewer
reserved bytes). It disturbs me that allocation/deallocation is being
spread around in so many different places. Sounds like an
anti-patterm :-(
Later VBO strategy, to save memory.
free user vertex memory after copying it to VBO. Keep only one user vertex array around?
Another VBO strategy to save time and even more memory
or use glMapBuffer() to get ptr to gpu meme, and put (newly computed) vertices directly in it, saving time ad memory.
And another (potential) speedup:
later, use a second vbo, with target GL_ELEMENT_ARAY_BUFFER to hold indices, usage GL_DYNAMIC_DRAW, or GL_STREAM_DRAW
Use the following to check for openGL errors:
#define CHECK_OPENGL_ERROR( cmd ) \
cmd; \
{ GLenum error; \
while ( (error = glGetError()) != GL_NO_ERROR) { \
printf( "[%s:%d] '%s' failed with error %s\n", \
__FILE__, __LINE__, #cmd, \
gluErrorString(error) ); \
}
To try and debig graphics, eliminate flushes, etc.
try mode=FL_SINGLE (seems to be double)
try mode=FL_FAKE_SINGLE (fake single using double. ???)
try setting damage(0) for Gl_windows. unset it later.
Forget that. I just removede the flushes and performance is way better, at least for VBOs on the mac.
AND it fixed the spinning screws up selection bugs (even without VBOs)
Here are the performance figures with all FL::flush() calls removed:
test is: ./vp -f b -i ~/data/stars/tycho.bin -r 5 -c 5
then seeing how long it takes in seconds to completely rerender 10 inverted selections (press 'i' 10 tiems fast at clock=0)
VBO=no, FAST_APPLE_VERTEX_EXTENSIONS=no 12 secs
VBO=no, FAST_APPLE_VERTEX_EXTENSIONS=yes 7.5 secs
VBO=no, FAST_APPLE_VERTEX_EXTENSIONS=yes 4.1 secs
Trying to put indices into a VBO, too. Hard freeze here on
tycho.bin. all points=0 on synthetic data. it seems that
indices_selected_packed or its mapping is messed up.
Debugging strategy: use indices_selected_packed as index array without
VBOS (i.e. using old vertex array)
Another performance idea (for OSX). Experiment with GL_APPLE_flush_render
http://developer.apple.com/graphicsimaging/opengl/extensions.html#GL_APPLE_flush_render
inside of FLTK?
do not waste too much time on this...... Need to get on to other
things.
23:21:07 28 Oct 06
Wasted all afternoon trying to get shared VBOs for indices (GL_ELEMENT_ARAY_BUFFER). It just doesn't work.
Maybe I should go back to using glColorPointer(), which should be able to be shared more easily. And not use glDrawElements().
And how should I stash this partially working code?
Need to pull out GL_ELEMENT_ARAY_BUFFER stuff, but leave in my other fixes.
15:09:13 29 Oct 06
Got GL_ELEMENT_ARAY_BUFFER VBOs working by using a separate VBO for indices from each selected set.
This reserves more space on te GPU thean it needs to, and probably more time.
There are probablty still crashing problems if one adds and removes plots, etc.
I'm going to check this in anyway, and back it out or fix it up later
Mon Oct 30 16:11:40 2006
1. Paul must fix RELOAD to use correct axes values. May have to call extract_and_redraw()
2. If easy, add a confirm dialog before over-writing files.
3. After paul makes these patches, we will make a new release.
4. And we will get him going with Makefile (for e.g. fltk-1.1.x) and win32 svn client.
11:22:25 31 Oct 06
TODO:
build script / build instructions
compile for powerpc
upload to trotsky
make tarballs for mac & linux versions
VBOS for indices
does glDrawElements() have to start from 0 when indices are in a VBO? Is this a GL bug?
does glBufferSubData() have to start from 0?
Or do multiple calls behave in an unexpected way?
VBOs to conserve main memory - can we do it
point sprites
hls for color selection
see bugs & ideas file
add back reading from stdin
code walkthrough
write paper
submit to arxiv, NASA TR
start opensource proceedure
write eric for viewpoints.org?
software of the year
presentation(s) at arc
presentations at fermilab, slac.....
FAQ
changelog ?
12:45:04 03 Nov 06
why does the linux version crash on xx.to ?
add "blend" button, default=on, which turns blending on/off.
allow different textures (sprites) for different selected sets.
add separate control of point appearance (hue, lum1, lum2, size,
symbol, alpha?...) for selected and deselected sets. Could have
"broadcast" switches (default=on, or "local", default=off) which
controls whether or not the parameter it controls are broacast to
(i.e. used by) the other windows. Panels could also have "protect"
switches which protect them from these broadcasts, similar to "lock
axis" protection. When the global panel is active, changes to the
selected set's appearance affect all selected points in all windows.
could decouple selections from windows: choose a "brush", select
anywhere, maybe choose another brush, select anywhere..... Could
optionally have windows remember the last brush they were using.
need to be able to selectively clear a window's selection without
clearing everything. This might be much easier with (per-window)
brushes - just reset a particular brush to have the characteristcs of
the non-selected points?
could optionally allow point size to scale with window transformation.
can optionally use glPointarameters() size scaling when third axis is
active.
Definitely have luminance scale with plot (optionally).
Have optional legend, (on lum, etc, sliders?) that is a fat line drawn
0,1,2...1000 times, on top of itself, shorter each time, with the
appropriate color, blending, etc.
can we use fog to change color based on "third" axis in 2D plots? 3D plots?
Ideas for paper: Good survey at:
wiki.eurovotech.org/pub/VOTech/DS6RelatedPublications/DimenReducAlgs.pdf
vp may excel at:
Data ink ratio = (data ink)/(total ink in the plot)
Data density = (number of entries in data matrix)/(area of graphic)
ideas from rank ordering (Schneiderman & Seo)
univariate orderings (to order axis in axis menu?)
Size of the biggest gap (0 to max range of dimensions):
Normality of the distribution (0 to inf):
Uniformity of the distribution (0 to inf):
The number of potential outliers (0 to n):
The number of unique values (0 to n):
Size of the biggest gap (0 to max range of dimensions):
kurtosis
skew, etc.
entropy, burges entropy, etc...
Bivariate orderings
pearson
entropy
# outliers
MI, etc, etc
# in "region of interest" (see below)
Chi^2 (see. p86 of ftp://ftp.cs.umd.edu/pub/hcil/Reports-Abstracts-Bibliography/2005-20html/2005-20.pdf)
scatterplot browser, esp. axis sliders. see fig. 3 (& test) of Seo & Schneiderman
/www.cs.umd.edu/hcil/multi-cluster/presentations/seo_shneiderman_rff_ivs.pdf
or browse in 2D ordering diplay (slide vert, slide horiz...)
3rd component?
after (or before) doing this, we can generalize to meta-scatterplot.
E.g. plot pearson vs. MI where each point represents a bivariate scatterplot. select in here.
10:16:24 05 Nov 06
Memory Performance puzzle: (OSX) Read in 10 million points, six vars. 2 windows. My calculations are:
theoretical: 10 M ps * (6+4) var/pt * 2 wd/var * 4 byte/wd + 2 window * 10 M vert/window * 4 wd/vert * 4 byte/wd = 800Mb + 320Mb = 1120Mb
but performance monitor (big top) shows: 1900Mb VM (1500 MB resident)
seems to be using 2-3 x as much memeory as it should. Bad. Why? Attempt to debug using
virtual resident
npoints nvars nplots predicted measured measured
10M 6 2 1120MB 1900MB 1500MB
1.1M 21 2 210MB 667MB 280MB
1.1M 21 25 600MB 953MB 953MB
checking with ObjectAlloc tool suggests ranked data is being kept
both per plot and globally. Let's check this. There are a lot of
global arrays, especially having to do with selection. Do we need
*all* of them? identity - do we need it?
OK - tried to eliminate the separate ranked data held be each & every
plot window, by replacing it with aliases tot he globally raked data,
which is kept around anyway. This seems to work. But memory s still high:
looks like 2M points are allocated, even when npoints<2M
virtual resident
npoints nvars nplots predicted measured measured
10M 6 2 1120MB 1700MB 1300MB
5M 6 10 1200MB 2250MB 1370MB
1M 6 50 900MB 2000MB 1300MB
1M 6 1 100MB 600MB 190MB
10M 6 1 1000MB 1400MB 873MB
Well, objectalloc says the last case is only using 900MB, so maybe the resident size is all that matters?
more investigations later.
One thing we coud do is eliminate the vertexes from plot_window
(when using VBOs). Just store them in their VBOs, and not in main
memory.
15:19:32 30 Nov 06
IDEA: when reading only one column, create a "line number" column. So that basic plotting works.
perhaps there should always be a line number column. (It does not need to be sorted).
(actually, no. It could be created, optionally, say via a cmd line argument. it should be
created automatically only when there is but a single column in the input data file.
14:09:41 04 Dec 06
FROM my todo list:
set up presentations
slac, arc, fermilab....
build script / build instructions
compile for powerpc
upload to trotsky
make tarballs for mac & linux versions
VBOS for indices
glDrawElements() have to start from 0?
does glBufferSubData() have to start from 0?
Or do multiple calls behave in an unexpected way?
Run time VBO check, cmd line arg.
VBOs to conserve main memory - can we do it?
More symbols.
Histogram filled. (w/alpha).
use hls for color selection
Brush object - has hls, color, symbol, size, footprint...
All points render using brush 0 initially
All windows start selecting using brush 1...
maybe not
Any window can change which brush it uses dynamically
Updates to brush properties are implicitly broadcast
need "overlap" (multiple selection) brush too
Windows also have additional brightness, bkg, size...
Windows need overplot button, default=on
3 modes for auto-clear selection:
always
when focus changes
when focus returns
never
"track cursor position" mode - prints numeric posn.
merge these w/bugs & ideas file
add back reading from stdin
code walkthrough
write paper
submit to arxiv, NASA TR
start opensource proceedure
write eric for viewpoints.org?
software of the year
open source approvals.
presentation(s) at arc
FAQ
man page
changelog ?
20:19:05 06 Dec 06
Use Flmm_tabs for tab bar - it allows an arbitrary number of tabs.
Make organized list of vp tasks/features/fixes.
17:45:19 03 Jan 07
Insstead of plotting x_i_k in column i vs. x_j_k in column j for each row k, create new "virttual" data file (perl?) containing
k i x_k_i and read that in.
12:09:50 04 Jan 07
Meetging w/Paul
priorities:
For intial release:
1) VBO command line argument
-- done
3) Fix sprites for selected only or remove them
4) Fix brightness sliders
4.5) window titles: "plot 4, pressure vs. temp"....
5) Documentation is accurate (readme, man page, about.html)
6) remove "line number" from asci/binary output
-- done
6.5) fix "sum vs. diff" noralization so data remains (approximately) centered in window.
-- done
7) Paul fix bugs in file chooser path
8) test
9) freeze, zip, and post
10) send out email
11) Schedule more talks/demos/marketing
12) Roses proposal
13) if possible, buttons in file dialog for: file type, selected/nonselected/all, line number/not, header modification, selection id.
14) if possible, pop up a confirm quit window when quitting, and when the user closes the control panel window, or the last plot window.
For "official" release
1) Paul one-click svn for windows
2) Paul break dependecy on devc++ devpack
3) fltk 2.0 / remove flews
4) Brushes
5) spreadsheet view?
6) some sort of save state
6) Fix window tiling to eleiminate big spaces between windows, make sure borderless works on mac.
6) mouse cursor tracking (option)
7) selection box dranw w/selection (brush) color
8) more file types
8) axis slider(s) (on plot windows?)
8) mutual info, corelation, etc, ranking in background, fed to change axes
9) axis pair(s) display(s), for selecting pairs based on MI, core, etc.
8) web site
9) freeze, test, zip, post, email.
10) software of the year.
10:54:41 18 Jan 07
"spin" control is too fast. Scale down rotation angle fro mrot slider before accumulating during spin.
11:47:35 05 Feb 07
ideas for histograms & densities:
1. make histogram stuff per axis
on/off, type (none, equiwidth, equidepth, ash...), nbins, scaling, coloring/filling
2. add new types (equidepth!)
3. create new variable (or "normalization"/transform): density estimate based on 1-D histogram
4. then allow user to form conditional estimates and joint estimates and product estimates.
5. Is conditional estimate P(y|x) something like the x-rank of a point within a particular y-bin?
In any case, that's what I want for voronoi volume of galaxy cluster normalized by distance.
We could do some nice ui things (in opengl windows) with Fl_Menu_Button::type(POPUP[123]).....