Amazon Glacier in Python with Boto (Part 2: Checking Inventory)

Step 1: Request contents in a vault.

from boto.glacier.layer1 import Layer1
import sys

access_key_id = "YOUR_ACCESS_KEY_ID"
secret_key = "YOUR_SECRET_KEY"

if len(sys.argv) < 2:
    print("Usage: python request_inventory.py VAULT_NAME\n")
    exit()

target_vault_name = sys.argv[1]
 
glacier_layer1 = Layer1(aws_access_key_id=access_key_id, aws_secret_access_key=secret_key)
 
print("operation starting...");
 
job_id = glacier_layer1.initiate_job(target_vault_name, {"Description":"inventory-job", "Type":"inventory-retrieval", "Format":"JSON"})
 
print("inventory job id: %s"%(job_id,));
 
print("Operation completed.")

The output may look like this:

{'Marker': None, u'RequestId': 'A_LONG_STRING', 'JobList': [{'CompletionDate': None, 'VaultARN': 'arn:aws:glacier:us-east-1:YOUR_AWS_ACCOUNT_ID:vaults/YOUR_VAULT_NAME', 'RetrievalByteRange': None, 'SHA256TreeHash': None, 'Completed': False, 'InventorySizeInBytes': None, 'JobId': 'A_LONG_STRING', 'ArchiveId': None, 'JobDescription': 'inventory-job', 'StatusMessage': None, 'StatusCode': 'InProgress', 'Action': 'InventoryRetrieval', 'ArchiveSHA256TreeHash': None, 'CreationDate': '2013-12-22T06:25:59.528Z', 'SNSTopic': None, 'ArchiveSizeInBytes': None}]}

Step 2 (optional): make sure that the job is being processed.

The code below check the status of all uncompleted jobs in a vault.

from boto.glacier.layer1 import Layer1
import sys

access_key_id = "YOUR_ACCESS_KEY_ID"
secret_key = "YOUR_SECRET_KEY"

if len(sys.argv) < 2:
    print("Usage: python check_jobs_in_a_vault.py VAULT_NAME\n")
    exit()

target_vault_name = sys.argv[1]
 
glacier_layer1 = Layer1(aws_access_key_id=access_key_id, aws_secret_access_key=secret_key)
 
print("operation starting...")

print glacier_layer1.list_jobs(target_vault_name, completed=False) # only uncompleted jobs will be printed

print("operation finished")

The result may look like this.

{'Marker': None, u'RequestId': 'A_LONG_STRING', 'JobList': [{'CompletionDate': None, 'VaultARN': 'arn:aws:glacier:us-east-1:YOUR_AWS_ACCOUNT_ID:vaults/YOUR_VAULT_NAME', 'RetrievalByteRange': None, 'SHA256TreeHash': None, 'Completed': False, 'InventorySizeInBytes': None, 'JobId': 'A_LONG_STRING', 'ArchiveId': None, 'JobDescription': 'inventory-job', 'StatusMessage': None, 'StatusCode': 'InProgress', 'Action': 'InventoryRetrieval', 'ArchiveSHA256TreeHash': None, 'CreationDate': 'A_TIME_STAMP', 'SNSTopic': None, 'ArchiveSizeInBytes': None}]}

Note that the "StatusCode" of the job is InProgress. After a few hours, the job might be done and its status will become like this:

{'Marker': None, u'RequestId': 'A_LONG_STRING', 'JobList': [{'CompletionDate': 'A_TIME_STAMP', 'VaultARN': 'arn:aws:glacier:us-east-1:YOUR_AWS_ACCOUNT_ID:vaults/YOUR_VAULT_NAME', 'RetrievalByteRange': None, 'SHA256TreeHash': None, 'Completed': True, 'InventorySizeInBytes': SIZE_IN_BYTES, 'JobId': 'A_LONG_STRING', 'ArchiveId': None, 'JobDescription': 'inventory-job', 'StatusMessage': 'Succeeded', 'StatusCode': 'Succeeded', 'Action': 'InventoryRetrieval', 'ArchiveSHA256TreeHash': None, 'CreationDate': 'A_TIME_STAMP', 'SNSTopic': None, 'ArchiveSizeInBytes': None}]}

Note that now the "StatusCode" is Succeeded and the job request is ready for retrieval.

Step 3: Fetch the result of this job (i.e., an inventory request).

from boto.glacier.layer1 import Layer1
import sys

access_key_id = "YOUR_ACCESS_KEY_ID"
secret_key = "YOUR_SECRET_KEY"

if len(sys.argv) < 3:
    print("Usage: python fetch_inventory.py VAULT_NAME JOB_ID\n")
    exit()

[target_vault_name, jobid] = sys.argv[1:3]
 
glacier_layer1 = Layer1(aws_access_key_id=access_key_id, aws_secret_access_key=secret_key)
 
print("operation starting...")

print glacier_layer1.get_job_output(target_vault_name, jobid) 
 
print("Operation completed.")

The result looks like this:

{u'TreeHash': None, 'VaultARN': 'arn:aws:glacier:us-east-1:YOUR_AWS_ACCOUNT_ID:vaults/MY_VAULT_NAME', u'ContentType': 'application/json', u'RequestId': 'A_LONG_STRING', u'ContentRange': None, 'InventoryDate': 'A_TIME_STAMP', 'ArchiveList': [{'ArchiveId': 'A_VERY_LONG_STRING', 'ArchiveDescription': 'THE_ARCHIVE_NAME', 'CreationDate': 'A_TIME_STAMP', 'SHA256TreeHash': 'A_LONG_HASH_VALUE', 'Size': A_BIG_NUMBER}]}

As you can see that I have only one archive in this vault.

Next time, I will talk about how to delete archives in a vault.

Apple, Google and Amazon show Mandela tributes but Microsoft only cares about holiday sales

On Thursday, Nelson Mandela left us. Since last night (Friday, Dec. 6), Apple,  Google and Amazon, has posted tributes on their homepage.

In particular, Apple used the same way they did when Steve Jobs passed away.

Micro$oft did nothing. It's just a holiday season to them.

Figure 1: Apple's US homepage, as captured on Dec. 7, 2013 (Source: Business Insider)

Figure 2: Google's homepage as captured on Dec. 7, 2013

Figure 3: Apple's Chinese homepage, as captured on Dec. 7, 2013

Figure 4: Amazon features Nelson Mandela Foundation on their website

Figure 5: Last but not least, Microsoft shows no care but holiday sales on its website.

High Frequency Brain Signals (HFBS) in EEG and MEG, Part Two: A scientific hypothesis

Last week, I wrote a blog post about High Frequency Brain Signals (HFBS) in EEG and MEG. In that post, I made an assumption that the HFBS we observed might be introduced by windowing function. Of course, I do not have any support to prove nor to disprove this assumption.

Today, I will explain why HFBS is a scientific hypothesis.

The signal in each  M/EEG (EEG or MEG) channel is the composition of activities of thousands or millions of neurons. Limited by the mechanism that neurons fire, one neuron won't generate HFBS - at least not those at hundreds of Hertz. When thousands or millions of them all fire, and the composition happens to be non-linear, things could be different.

For example, from basic trigonometry, we know that:
$$\sin(2x) = 2\sin(x)\cos(x)$$ and $$\cos(2x) = \cos^2(x) - \sin^2(x)$$
See? The frequency is doubled on the left hand side of the two equations.

If multiplication between brain signals could happen, then we can get signals of higher frequency. In physics and electrical engineering, such higher frequency signals are called harmonics and are related to a very annoying trouble in communication systems: intermodulation noise/distortion. Can you guarantee that our brain is a linear system?

Since neurons do not fire independently (many papers are discussing synchrony and connectivity of the brain), it's reasonable to assume that multiplication and even more complicated operations could happen on signals from neurons. Such composed signals are captured by M/EEG.

Therefore, it is scientific to hypothesize that HFBS is real from the brain, not introduced by sampling techniques. In this sense, HFBS is a useful tool to study how our brain works.

Comments are welcome, especially those from DSP or neuroscience field.

OpenKinect and Kinect Model 1473

Since I am a computer engineering professor now, I do something other than computational neuroscience (neuroinformatics) or bioinformatics.

Recently, I started working on a project using 4D camera (RGB + depth). I spent $30 on eBay to get a Kinect. Today I just got it. First, I decided to use OpenKinect/freenect to make sure that the depth sensor is working. 

My Kinect is a late version 1473. The freenect package comes with my Ubuntu Linux 13.04 is too old - though this version of Kinect has been around for quite a while.

I struggled a lot and couldn't get it to work, until I saw this in freenect's discussion.

If you have Kinect 1473, go to OpenKinect's Github repository for the latest version. Then follow the instruction on their wiki to compile and install. Then it should work.

Oh, here is my first encounter with Kinect. Pay attention to the wrinkles on my shirt and the depth hole in my mouth - I intentionally did that.

Also I found out that open source drivers do not work for Kinect for Windows. Be sure to get Kinect for XBox.

See also:

• http://pl8824.blog.com/ubuntu-and-kinect/

High Frequency Brain Signals (HFBS) in EEG and MEG, Part One: A delusion from spectral leakage?

Update 10/23/2013: If the sampling frequency is 2000Hz, the highest frequency that windowing function can introduce is 1000Hz. If 1000Hz signal is observed in digital signal, it does not necessarily mean that the analog signal has 1000Hz component. In this case, the 1000Hz signal may not be from the brain but the sampling technique used.

About a month ago, I met a friend in Cincinnati Children's Hospital on my way from Texas to Ohio. His research is about high frequency brain signals (HFBS, >70Hz) in MEG. He told me that many reviewers couldn't quite understand why there are signals up to hundreds of Hertz in MEG because neuronal activities cannot be that fast.

As a man of science, I strongly believe in my friend's hypothesis but at the same time remain equally skeptical.

Hence, I decide to write 2 blog posts to first challenge him and then support him. Today, I am gonna give him some hard time: it is scientifically reasonable that HFBS may be a delusion caused by a phenomenon known as spectral leakage. 

Short version: One can get high amplitude at any frequency (between 0Hz and half of the sampling frequency) from even a constant signal (i.e., exactly 0Hz) - if you use a bad window function when sampling the data.

Here is the detail. Today, almost all neurophysiological signals, including EEG and MEG, are sampled using Digital Signal Processing (DSP). In order to digitize data from the analog world, we sample them by applying a window function. A bad window function, e.g., a rectangular window, can give you high amplitude from 0Hz to half of the sampling frequency in spectral domain, like this:

Rectangular window and its Fourier Transform. Leaked frequencies are relatively high.
Source: Wikipedia (You will see the same in any DSP textbook.)

This phenomenon is called spectral leakage. You can consider it as frequency components not in the signal "leak" into the spectrum.

Using a better window function, such as Hamming window, can reduce the effect of spectral leakage. For example, the amplitude(s) of "real" frequency(-ies) can be much higher than those of leaked frequencies.

Rectangular window and its Fourier Transform. Leaked frequencies are 1000 times (-60dB) lower than the main frequency. Source: Wikipedia (You will see the same in any DSP textbook.)

But sadly we cannot eliminate spectral leakage. Hence, you always have leaked frequencies while their amplitudes may not be small. The question is, are the amplitudes of HFBS high enough? I need to read more papers on this topic before I can make a claim.

If you wanna understand mathematically why we can never get rid of spectral leakage, go grab any DSP textbook (unfortunately, lots of math) and you will get it.

Comments are welcome, especially those from DSP or neuroscience field.

References:

• Xiang et al., Neuromagnetic correlates of developmental changes in
  endogenous high-frequency brain oscillations in children: A wavelet-based beamformer study, Brain Research, 1274 (2009):28--39
  [The discussion part of this paper explains the sources of low frequency neuronal activities and high frequency ones. ]
• Xiang et al., Noninvasive localization of epileptogenic zones with ictal high-frequency neuromagnetic signals, Journal of Neurosurgery: Pediatrics,
  2010 Jan; 5(1):113--22.