Reading parts from a text file

[tedor2]

Dear SC users,

I am wondering whether someone could give me some hints how I could use an exported text file in SC. Basically I measure brainwave activity in a software called IBVA. The software is able to export the results in many ways (FFT, power, ...). Now, I know most things are possible is Supercollider.

I attache a screenshot of IBVA and the file itself.

http://www.tedor2.extra.hu/mix/screenshot.jpg

http://www.tedor2.extra.hu/mix/exported_data

The file is a bit difficult to read:

Two channels are separated (Left CH1 and Right CH2 /starts around the middle of the text).

After "Lines           Sampling     Relative Time" the time is shown where a value got written below (both channels are measured at the same time).

00000000   00000000    00:00:00.000

00000001   00000128    00:00:01.066

00000002   00000256    00:00:02.133

after these numbers are the ones I would like to use in relation with the time above.

Is this possible to read these data into something and then use them to be read into Synth?

Any help appreciated,
all the best
krisztian

[dewdrop_world]

Reading a file like this takes a few basic capabilities that most computer languages have.

- reading strings from a file
- searching for characters or substrings in a string
- extracting substrings
- converting strings to numbers

Supercollider can do all of these things, so yes, of course it's possible.

What you need to do is think step-by-step about the parts of the file and how each should be processed.

First is a header, which you just want to ignore up until the line that says "Left CH1." That's pretty easy -- read a line and check to see if it == "Left CH1" -- if so, stop and go onto the next section. The standard way to do this is with a while loop. (Checking for notNil is also important because you want the loop to stop if the file runs out of lines.)

Code:

var file = File(path, "r"),
line;

line = file.getLine;
while { line.notNil and: { line != "Left CH1" } } {
line = file.getLine;
};

// the above line = ...; while ...; could be written more concisely as:
while { (line = file.getLine).notNil and: { line != "Left CH1" } };

Then follow the data. What steps do we need for each line?

- there are some bad characters; strip them out
- split the string wherever there is white space
- remove any empty substrings in the partitioned string (just in case)
- throw out the first three columns
- convert the rest to floating-point numbers

Oh, and stop when a line doesn't contain any numbers.

Code:

while { (line = file.getLine).notNil and: { line.any(_.isDigit) } } {
// remove bad characters
line = line.reject({ |ch| ch.ascii > 127 })
// split wherever there is a space (including tabs)
.delimit({ |ch| ch.isSpace })
// remove empty strings
.reject({ |substr| substr.size == 0 });
// starting with the 4th column, convert each string to a float
// the result is an array of the FFT magnitudes from the file
matrix = matrix.add(line[3..].asFloat);
};

Then skip a few lines until you get to "Right CH2" and do the same thing to read the next table.

I'm not sure of the format you need for the final matrix, but array manipulations can certainly get you there.

Solving problems like this is a matter of thinking it through, one step at a time. Looking at the file as a whole, it seems difficult indeed but there is a very clear action to take for each separate line. So look at one line, and break down the process into simple steps.

Also it's a good idea to use protect when processing file data. If there is an error in the middle of file processing, you could leave a file handle open and then have no way to close it, not so good.

General template:

Code:

file = File(path, "r");
if(file.isOpen) {
protect {
... do file stuff in here ...
} {
file.close;
};
} { "Couldn't open file".warn; };

With this, even if an error occurs inside the protect block, the file is always closed.

James

[tedor2]

Wow,

James, I will do my best!

Thank you very much for your time!

Krisztian

[tedor2]

Dear James,

I rely on your help. Step by step...

I think I understood that a text file is made up of lines. What we want to do is to get the values in a matrix so we can easily access them. 2d matrix x = fft numbers; y = time.

Code:

(

var file = File("/Users/krisztianhofstadter/Desktop/EEG/SC\ read/exported_data", "r"),
line;



line = file.getLine;
while { line.notNil and: { line != "Left CH1" } } {
line = file.getLine;
};

With the first part of the code we read a whole text file into a variable (file) and set up a new variable (line).

The second part is trickier:

line = file.getLine;

"getLine = Reads and returns a String up to lesser of next newline or 1023 chars."
I got confused here:)

The while function has two parts:

testFunc :{ line.notNil and: { line != "Left CH1" } }
bodyFunc : {line = file.getLine;}

"While" starts reading the text from the beginning (testFunc). It loops the "testFunc" until  it's false. In scope > This mean that the "bodyFunc" evaluates "line = file.getLine", where this evaluation (line = file.getLine) reads strings from the text lines in a loop till it is false?
 
getLine reads strings from the file >

are these strings the "text lines" them self?
or are these strings characters next to each other like a word or numbers (Left ~ 000)
or just one character in the text (1 ~ L)
or is "Left CH1" a string?

.... here the while loop stops in front of the next string? section? character?

If "Left CH1" is a string, while was looking for to stop, I think I am alright so far.

James, please let me know what you think so far of my progress, your help is appreciated!

All the best,
Krisztian

[dewdrop_world]

Dear James,

I rely on your help. Step by step...

I think I understood that a text file is made up of lines.

A text file is really just a stream of bytes. Some of those bytes are interpreted to mean the end of a line (but it's not required to read the file that way).

Code:

(

var file = File("/Users/krisztianhofstadter/Desktop/EEG/SC\ read/exported_data", "r"),
line;



line = file.getLine;
while { line.notNil and: { line != "Left CH1" } } {
line = file.getLine;
};

With the first part of the code we read a whole text file into a variable (file) and set up a new variable (line).

No -- file = File(path, mode) doesn't read anything. It just opens a file handle. You can read data from the file handle - or just close the handle without reading anything (which is a way to test for a file's existence on disk, e.g. -- see File.exists in the SC lib).

The second part is trickier:

line = file.getLine;

"getLine = Reads and returns a String up to lesser of next newline or 1023 chars."
I got confused here:)

The while function has two parts:

testFunc :{ line.notNil and: { line != "Left CH1" } }
bodyFunc : {line = file.getLine;}

"While" starts reading the text from the beginning (testFunc). It loops the "testFunc" until  it's false. In scope > This mean that the "bodyFunc" evaluates "line = file.getLine", where this evaluation (line = file.getLine) reads strings from the text lines in a loop till it is false?
 
getLine reads strings from the file >

are these strings the "text lines" them self?
or are these strings characters next to each other like a word or numbers (Left ~ 000)
or just one character in the text (1 ~ L)
or is "Left CH1" a string?

Try it for yourself -- running some exploratory code will teach you more about reading files.

f = File(path, "r");
f.getLine;   // what do you see in the post window?

f.getLine.dump;  // proves the result is a String, and shows some of the contents

In the source file, there is a line "Left CH1" that occurs at the beginning of the first data section. When that line is read, line != "Left CH1" becomes false and the while loop stops. The file's position should be after the newline character terminating "Left CH1" (i.e., the next character to be read will be the first character on the line following "Left CH1").

If "Left CH1" is a string, while was looking for to stop, I think I am alright so far.

Yes. Double-quote syntax for string literals is explained in the Literals help file.

James

[tedor2]

Dear James,

thank you for clearing the haze! Do you think I should try learning C? Or which would be the best way to learn the language? Are the help files enough?

There is one major issue concerning this project. There are some invisible characters in the text file:

http://www.tedor2.extra.hu/mix/scpostmicro.jpg

Therefore it stops when it reaches the first inverted question mark!
Hmm... shall I try to get rid of this character somehow and than start?

Thank you,
all the best
Krisztian

[dewdrop_world]

thank you for clearing the haze! Do you think I should try learning C? Or which would be the best way to learn the language? Are the help files enough?

Nah, don't waste your time with C. (C is still useful for a lot of things, but most of the programming world has moved on to C++ or Java. C is not object-oriented, so it won't teach you much about SC.)

I can't think of a "general programming concepts for SC users" guide... maybe look at the Ruby language. It is somewhat similar to SC in some ways.

There is one major issue concerning this project. There are some invisible characters in the text file:

http://www.tedor2.extra.hu/mix/scpostmicro.jpg

Oh, this is interesting. Yeah, you sometimes have to dig into the raw data to see what's really there before writing the final code. Sometimes I do this to look at the raw data:

Code:

f = File(p, "r");
z = Int8Array.newClear(200);
f.read(z);
z.clump(8).do({ |row|
row.do({ |item|
Post << item.asHexString(2) << " ";
});
"\t".post;
row.do({ |item|
item = item.asAscii;
Post << if(item.isPrint) { item } { "." };
});
Post << $\n;
});

f.close;

// prints:
0D 54 69 74 6C 65 20 6F    .Title o
66 20 74 68 65 20 45 78    f the Ex
70 6F 72 74 20 42 72 61    port Bra
69 6E 20 44 61 74 61 3A    in Data:
09 09 09 45 78 70 6F 72    ...Expor
74 20 42 72 61 69 6E 20    t Brain
44 61 74 61 0D 0D 4D 61    Data..Ma
78 20 62 75 66 66 65 72    x buffer
20 73 69 7A 65 3A 09 09     size:..
09 20 30 31 3A 30 30 3A    . 01:00:
30 30 2E 30 30 30 0D 53    00.000.S
74 61 72 74 20 54 69 6D    tart Tim
65 20 69 6E 20 62 75 66    e in buf
66 65 72 3A 09 09 09 20    fer:...
30 30 3A 30 30 3A 30 30    00:00:00
2E 30 30 30 0D 45 6E 64    .000.End
20 54 69 6D 65 20 69 6E     Time in
20 62 75 66 66 65 72 3A     buffer:
09 09 09 20 30 30 3A 30    ... 00:0
32 3A 34 37 2E 35 39 31    2:47.591
0D 0D 49 42 56 41 34 20    ..IBVA4
61 6E 64 20 63 75 72 72    and curr
65 6E 74 20 66 69 6C 65    ent file
20 69 6E 66 6F 72 6D 61     informa
74 69 6F 6E 3A 09 09 09    tion:...

The first thing I see from this is that 0D represents a line break in this file, and getLine doesn't recognize it as a line break. (How do I know this is the line break? By comparing the raw output to the appearance in a text editor.)

The bad characters show up as NUL in notepad++ on my windows machine, meaning ascii code 0. That is a fairly serious data problem that is harder to correct in SC. Since SC's underbelly is C, the C-style rule for strings apply where a 0 byte means the end of the string. Probably the string in memory has all the characters, but they are ignored past 0 because of the C rule.

SC doesn't have a built-in method to strip 0 bytes, so you'll have to write your own getLine function.

Code:

~getLine = { |file, endChar(13.asAscii)|
var char = file.getChar, str = String.new;
if(char.isNil) { nil } {
while { char.notNil and: { char != endChar } } {
if(char.ascii > 0) { str = str.add(char) };
char = file.getChar;
};
};
str
};

Sorry this is such a pain... working with files you didn't create yourself is sometimes hard.

James

[tedor2]

dear James,

there is one good news: I emailed the manufacturer the "inverted question mark" issue, he updated the software and magic: there are no question marks. At least I got closer...

I also tried FileReader and TabFileReader. They seem to get the numbers in arrays as well. Is the matrix at the end of your code something like this as well?


thank you for the codes below, they helped me to explain the manufacturer what the problem is!
best

Krisztian

[tedor2]

Dear James,

1, I am ok,  if I use a text file with just the lines of useful data "simple_lines16.txt". (copy and paste from the original). TabFileReader seems to make things easier for a dummy like me. I still would like to use the original file "leftandright16ch.txt", where both channels (Left CH1 and Right CH2) could be read in arrays.

Well, I think I just have to find out how to get the line number of where "Left CH1" and "Right CH2" is positioned. Then I could add and minus the numbers, and get the array to be read in from these new parameters. Do you have an idea how to get this line position?

http://tedor2.extra.hu/mix/ReadintoArray.zip

2, I wanted to send you an email on your site concerning the CD with your music, there is an error message after clicking on send. I wonder whether you would send it to the UK as well? ($?)

Thank you in advance,
Best
K

[dewdrop_world]

Dear James,

1, I am ok,  if I use a text file with just the lines of useful data "simple_lines16.txt". (copy and paste from the original). TabFileReader seems to make things easier for a dummy like me. I still would like to use the original file "leftandright16ch.txt", where both channels (Left CH1 and Right CH2) could be read in arrays.

Well, I think I just have to find out how to get the line number of where "Left CH1" and "Right CH2" is positioned. Then I could add and minus the numbers, and get the array to be read in from these new parameters. Do you have an idea how to get this line position?

http://tedor2.extra.hu/mix/ReadintoArray.zip

The easiest way to find the line numbers is to use a proper text editor (not the one built into sc.app). A lot of programmers swear by text mate, but there are a bunch of freeware choices. Many editors, especially programming-oriented ones, have the option of showing line numbers in the left margin.

But, looking at the tab file reader class definition, I don't see an easy way to tell it to stop after a given number of lines. I don't have a quick answer to that problem (except that I probably would write my own reader and parser using the outline in my earlier post, especially if this is something I expect to do a lot).

2, I wanted to send you an email on your site concerning the CD with your music, there is an error message after clicking on send. I wonder whether you would send it to the UK as well? ($?)

Sure, I'll send to the UK. What was the error message? I can't help you much with the error if I don't know what the error is. (Unfortunately this is all too common in the computer world -- people think it's enough to say "there was an error." So then it takes longer to solve the problem because it's necessary to do another back-and-forth to find out what the problem really was. Sorry to go on about this -- it is a major peeve of mine.)

I can tell you that I've had horrible problems with spam from that page, so I block actually rather a lot of phrases. I also had to put in a timer because a lot of spammers submit the input form instantaneously after visiting the page. The side effect is that if you write the message to me in another editor, then go to my page, paste it in quickly and hit "send," as far as the website is concerned, you look like a spammer. Try waiting at least 15 seconds before sending.

Or, if all else fails, send me a PM here.
James