First time here? Check out the FAQ!

Ask Your Question
2

Derivative of a CSV data set

asked 5 years ago

EnlightenedFunk gravatar image

updated 5 years ago

I am trying to get the derivative of this CSV data set, and this is code I have so far I thought it would be similar to this matlab code:

dy=diff(y)./diff(x)
plot(x(2:end),dy)

Data Set:

0,2.58612
0.00616025,2.20018
0.0123205,1.56186
0.0184807,0.371172
0.024641,0.327379
0.0308012,0.368863
0.0369615,0.322228
0.0431217,0.171899
0.049282,-0.0635003
0.0554422,-0.110747
0.0616025,0.0701394
0.0677627,0.202381
0.073923,0.241264
0.0800832,0.193697
0.0862434,0.0797016
0.0924037,0.0103144
0.0985639,0.096153
0.104724,0.216782

There's more data...a lot more This is my attempt at the code:

import csv
data = list( csv.reader(open('C:/images/TEST1.txt','rU')) )
data = map(lambda x: [float(x[0]),float(x[1])],data)
P = list_plot(data, plotjoined= True, color = 'blue', xmin = 0, xmax = 3)
TeraHertz = text('Tera Hertz', (1.5,-.9)) 
Absorbance = text('Absorbance', (0.8,5))
g = P + TeraHertz + Absorbance
g.show()
latex(g)
g.save('TeraHertzTEST1.pgf')
t=diff(x[1]).diff(x[0])
l = plot(x[0],t)
l.show()
Preview: (hide)

Comments

If possible, please post a link to the csv data set. It might help come up with answers to the question.

slelievre gravatar imageslelievre ( 5 years ago )

@slelievre Ok I will, it keeps saying greater thaan 60 points to upload file....

EnlightenedFunk gravatar imageEnlightenedFunk ( 5 years ago )

@slelievre I entered as many data points there is just a lot more data.

EnlightenedFunk gravatar imageEnlightenedFunk ( 5 years ago )
1

What actually is the problem? Why didn't this work? If the data is very large you might also consider using numpy to load the data set: import numpy as np; data = np.loadtxt('data.txt', delimiter=','). See loadtxt for more. For large data sets using Numpy will be both much faster than manually parsing strings returned by the standard Python csv module, and will be much, much more memory-efficient. Honestly, the SageMath documentation does not do enough to emphasize use of Numpy.

Iguananaut gravatar imageIguananaut ( 5 years ago )

I think that the problem was the lack of a pure Python equivalent of the Matlab diff function, which can be replaced with a list comprehension. Anyway, you are right, things are more easily and efficiently done with Numpy.

Juanjo gravatar imageJuanjo ( 5 years ago )

1 Answer

Sort by » oldest newest most voted
4

answered 5 years ago

Juanjo gravatar image

updated 5 years ago

Please, try the following code:

import csv
with open("test.csv") as csv_file:
    data = list(csv.reader(csv_file))
ndata = len(data)
data = map(lambda x: [float(x[0]),float(x[1])], data)
data = matrix(data)
P = list_plot(data, plotjoined=True, color="blue",
              frame=True, axes=False, axes_labels=["Tera Herz", "Absorbance"])
P.show()
t = [(data[i+1,1]-data[i,1])/(data[i+1,0]-data[i,0]) for i in range(ndata-1)]
l = list_plot(zip(data[1:,0].list(),t), plotjoined=True, color='red',
              frame=True, axes=False, axes_labels=["Tera Herz", "Derivative"])
l.show()

Here, test.csv is a file with the data you provided. As an alternative, with a more clear code but perhaps slower:

import csv
with open("test.csv") as csv_file:
    data = csv.reader(csv_file)
    xx, yy = [], []
    for row in data:
        xx.append(float(row[0]))
        yy.append(float(row[1]))
ndata = len(xx)
P = list_plot(zip(xx,yy), plotjoined=True, color="blue",
              frame=True, axes=False, axes_labels=["Tera Herz", "Absorbance"])
P.show()
t = [(yy[i+1]-yy[i])/(xx[i+1]-xx[i]) for i in range(ndata-1)]
l = list_plot(zip(xx[1:],t), plotjoined=True, color="red",
              frame=True, axes=False, axes_labels=["Tera Herz", "Derivative"])
l.show()

Edited. Following the advice of @Iguananaut, let us use Numpy. The code becomes:

import numpy as np
data = np.loadtxt("test.csv", delimiter=",")
P = list_plot(data, plotjoined=True, color="blue",
              frame=True, axes=False, axes_labels=["Tera Herz", "Absorbance"])
P.show()
t = np.diff(data[:,1])/np.diff(data[:,0])
l = list_plot(zip(data[1:,0],t), plotjoined=True, color='red',
              frame=True, axes=False, axes_labels=["Tera Herz", "Derivative"])
l.show()

In any case I get the following pictures: image description image description

Preview: (hide)
link

Comments

1

For actually loading the csv data I would use Numpy and not these slow pure Python loops. See my comment above.

Iguananaut gravatar imageIguananaut ( 5 years ago )

Hello this might seem annoying but is there a way to export that derivative data into a CSV?

EnlightenedFunk gravatar imageEnlightenedFunk ( 5 years ago )
1

Yes, just add

np.savetxt("deriv.csv", zip(data[1:,0],t), delimiter=",")

at the end of the last block of code. See https://docs.scipy.org/doc/numpy/reference/generated/numpy.savetxt.html for details on savetext.

Juanjo gravatar imageJuanjo ( 5 years ago )

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 5 years ago

Seen: 1,138 times

Last updated: Jun 12 '19