[rrd-users] Neuro data in RRD - ns time steps?

permalink · #1 (**permalink**) 08-28-2007

--===============1501869891==
Content-Type: multipart/alternative; boundary=Apple-Mail-2--443111008

--Apple-Mail-2--443111008
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed

Hi All,

I'm an academic neuroscientist who is looking for an efficient way to
store and retrieve recorded neuronal data for use in an intranet data
browser app. After an initial read through the documentation and
tutorials, RRD seems to be very close to what I had been thinking of
with one exception, our data is recorded at 5-25KHz, so we need time
steps of 50-200 nanoseconds. So I am wondering... has anyone modified
RRD to work with smaller time steps? If not, I am curious about how
much coding would be required to do this (i.e. how deeply embedded is
the time structure currently used in the database)

My alternatives would be to use a BLOB system in MySQL, or just use
file references in a database and build a custom system to scan the
binary data files and return the requested chunks. So I would also
be curious if anyone has any opinions (performance, scalability,
etc.) of different ways of building time series data browsers.

By the way, our files are large - each experiment is recorded on 60
channels for 2 hours, which we break into consecutive 2 minute files
(~ 70Mb for a 5Khz recording) for ease of use. I would ideally like
to have an RRD for each 2 hour experiment, with each channel as one
GAUGE DS containing 36-180M data points (in the native format theses
are 2 byte unsigned integers). Are these sorts of numbers worrisome
in terms of performance?

Thanks,
Peter

Peter Passaro
Research Officer
Dept of Informatics/Dept of Biology
Pevensey III
University of Sussex
Falmer, Brighton BN1 9QH, UK
Office + 44 (0)1273 877590
P.A.Passaro@sussex.ac.uk

--Apple-Mail-2--443111008
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=ISO-8859-1

<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; ">
Hi All,<div><br class=3D"webkit-block-placeholder"></div><div>I'm an =
academic neuroscientist who is looking for an efficient way to store and =
retrieve recorded neuronal data for use in an intranet data browser app. =
After an initial read through the documentation and tutorials, RRD seems =
to be very close to what I had been thinking of with one exception, our =
data is recorded at 5-25KHz, so we need time steps of 50-200 =
nanoseconds. So I am wondering... has anyone modified RRD to work with =
smaller time steps? If not, I am curious about how much coding would be =
required to do this (i.e. how deeply embedded is the time structure =
currently used in the database)</div><div><br =
class=3D"webkit-block-placeholder"></div><div>My alternatives would be =
to use a BLOB system in MySQL, or just use file references in a database =
and build a custom system to scan the binary data files and return the =
requested chunks. =A0So I would also be curious if anyone has any =
opinions (performance, scalability, etc.) of different ways of building =
time series data browsers.=A0</div><div><br =
class=3D"webkit-block-placeholder"></div><div>By the way, our files are =
large - each experiment is recorded on 60 channels for 2 hours, which we =
break into consecutive 2 minute files(~ 70Mb for a 5Khz recording) for =
ease of use. I would ideally like to have an RRD for each 2 hour =
experiment, with each channel as one GAUGE DS containing 36-180M data =
points (in the native format theses are 2 byte unsigned integers). Are =
these sorts of numbers worrisome in terms of performance?</div><div><br =
class=3D"webkit-block-placeholder"></div><div>Thanks,</div><div>Peter=A0</=
div><div><br><div> <span class=3D"Apple-style-span" =
style=3D"border-collapse: separate; color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0; "><span class=3D"Apple-style-span" =
style=3D"border-collapse: separate; border-spacing: 0px 0px; color: =
rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: =
normal; font-variant: normal; font-weight: normal; letter-spacing: =
normal; line-height: normal; text-align: auto; =
-khtml-text-decorations-in-effect: none; text-indent: 0px; =
-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
white-space: normal; widows: 2; word-spacing: 0px; "><div =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><div style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">Peter Passaro</div><div =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Research Officer</div><div style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Dept of =
Informatics/Dept of Biology</div><div style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Pevensey =
III</div><div style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">University of Sussex</div><div =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Falmer, Brighton BN1 9QH, UK</div><div =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Office + 44 (0)1273 877590</div><div =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; "><a =
href=3D"mailto:P.A.Passaro@sussex.ac.uk">P.A.Passa ro@sussex.ac.uk</a></div=
></div><br class=3D"Apple-interchange-newline"></span></span> =
</div><br></div></body></html>=

--Apple-Mail-2--443111008--

--===============1501869891==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

--===============1501869891==--

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode