How to add pre-commit hook for JIRA tracking in git commits.

Having a clean and useful commit messages always makes debugging easier. There are many different patterns people follow to maintain neat git log history. Here is what I like the most, I always like to link a JIRA issue id with commit, so that any point in time, I can check the description of the task, for which the code change was made.
    To make it compulsory, so that other committers will also follow the same commit message convension, you can add a pre commit hook in your git repository.
Steps to add hook:
1) cd into your git repository folder.
2) Since this is commit-msg hook, you'll need to make it executable.

chmod a+x .git/hooks/commit-msg

3) Now update the commit-msg file with the following content.

 test "" != "$(grep 'JIRA-' "$1")" || {
       echo >&2 "ERROR: JIRA issue number missing in commit message."
       exit 1;
}

Here replace your project name with JIRA.

   Just to validate its working,try to make a commit without the pattern (JIRA-) , You should get the error "ERROR: JIRA issue number missing in commit message."
Enjoy git committing :) :) 

How to enable date timestamp in bash history.

Many times while debugging I have a question, when did I execute this command? Here is a way to enable date and timestamp while listing your bash history.

Without date and timestamp, your history will look like

chetna.chaudhari@Chetna:~$ history
ps aux
jps
ls
clear
history

To enable date and timestamp,

chetna.chaudhari@Chetna:~$ export HISTTIMEFORMAT='%F %T '

Here, %F enables date in yyyy-mm-dd format (%Y-%m-%d) %T enables time in hour:minutes:seconds (%H:%M:%S) So now your history should look like

chetna.chaudhari@Chetna:~$ history
1  2015-04-08 19:49:35 ps aux
2  2015-04-08 19:49:35 jps
3  2015-04-08 19:49:35 ls
4  2015-04-08 19:49:35 clear
5  2015-04-08 19:49:36 ps aux | grep sshd
6  2015-04-08 19:49:37 history

To make it permanent add the export command to your .bash_profile file.

HDFS - Quota Management

HDFS Quotas:

You can set two types of quotas in HDFS

a. Space Quota: The amount of space used by given directory

b. Name Quota: The number of file and directory names used.

Notes:

• Quotas for space and names are independent of each other
• File and directory creation fails if creation would cause the quota to be exceeded.
• Block allocations fail if the quota would not allow a full block to be written.
• Each replica counts against quota. For eg. if user is writing 3GB file with replication factor of 3, then 9GB will be consumed from his quota.
• Largest quota is Long.Max_Value

HDFS Quota Operations:

a. Set a Name quota:

ACL: Only admin can perform this operation.

Command: Hadoop admin can use following command to set name quota.

hadoop dfsadmin -setQuota number_of_files path

eg.

hadoop dfsadmin -setQuota 100 /grid/landing

Explanation: It sets hadoop quota to 100, which means user can create 100 files including directories under /grid/landing path.

b. Clear a Name quota:

ACL: Only admin can perform this operation.

Command: Hadoop admin can use following command to clear name quota.

hadoop dfsadmin -clearQuota path

eg.

hadoop dfsadmin -clearQuota /grid/landing 

c. Set Space quota:

ACL: Only admin can perform this operation.

Command: To set space quota, hadoop admin can use following command,

hadoop dfsadmin -setSpaceQuota size path

eg.

hadoop dfsadmin -setSpaceQuota 15G /grid/landing

Explanation: It means user can write upto 5GB ( 5 * 3 = 15) of data under /grid/landing path , assuming the replication factor of 3. Here user cannot write data less than block size. why? Because HDFS assumes an entire block will be filled, when its allocated. eg. say, if some path /projects/ingestion has quota of 50 MB, and if someone is writing a file of 10MB under this path, it'll fail, because of quota violation. Here HDFS thinks user is writing 384MB (128 * 3) data, instead of (10 * 3 = 30 MB).

d. Clear Space quota:

ACL: Only admin can perform this operation.

Command: To clear space quota, admin can use following command,

hadoop dfsadmin -clearSpaceQuota path

eg.

hadoop dfsadmin -clearSpaceQuota /grid/landing

e. Get quota allocation of Path:

ACL: Anyone can check quota allocation of path

Command: Hadoop admin can use following command to check quota allocation of a path,

hadoop fs -count -q path

eg.

hadoop fs -count -q /grid

Explanation: Above command will give output of following format,

hadoop fs -count -q /grid
9223372036854775807 9223372036854775333 none  inf  141  333 655855032 /grid

where ,

column1 --> Namespace quota, which means total 9223372036854775807 files can be created
column2 --> Available Namespace quota, user can add 9223372036854775333 files .
column3 --> Space quota
column4 --> Available space quota
column5 --> Number of directories
column6 --> Number of Files
column7 --> Size of content available
column8 --> Path