Setting up PHP-Kafka using librdkafka wrapper phpkafka from EVODelavega under Ubuntu.

We will install kafka for PHP based on the libraries:

https://github.com/EVODelavega/phpkafka
https://github.com/edenhill/librdkafka

Setting up
==========
Fast way (Only Ubuntu)
--------

$ sudo apt-get install librdkafka1

Dependencies
------------
First let install some dependencies (installing manually)

$ sudo apt-get install libsasl2-dev liblz4-dev

Installing php-rdkafka (Other OS)
----------------------

$ git clone https://github.com/edenhill/librdkafka/
$ cd librdkafka
$ ./configure
$ make
$ make test
$ sudo make install

Installing phpkafka Extension
-----------------------------

$ git clone https://github.com/EVODelavega/phpkafka.git
$ cd phpkafka
$ phpize
$ ./configure --enable-kafka
$ sudo make install

If you want to check that kafka is in place:

$ php -m | grep kafka
kafka


Final PHP configuration
-----------------------
1) Create a PHP file containing only the call to phpinfo()

2) Run the script from your browser and check the path of the config ini files that your system is using. For example, my additional INI files are in /etc/php5/fpm/conf.d.

3) Creates a new file librd.con in that folder. Example:

$ sudo touch /etc/php5/fpm/conf.d/librd.conf

4) Add to the file the path of your libraries: /usr/local/lib. You can also do (as superuser):

$ echo "/usr/local/lib" >> /etc/ld.so.conf.d/librd.conf

6) Create the file 20-kafka.ini in your additional INI directory:

$ sudo vi /etc/php5/fpm/conf.d/20-kafka.ini

and insert the line:

extension=kafka.so

7) Run the command

$ sudo ldconfig

8) Check that everthing is file and the extension was loaded

$ ldconfig -p | grep kafka

9) Reset services

sudo /etc/init.d/php5-fpm restart


Setting the kafka with Docker for testing
=========================================

1) Download and install the containers (Check https://hub.docker.com/r/confluent/platform/)

$ docker-machine start
$ eval $(docker-machine env)
$ docker pull confluent/platform

2) Run the hub

# Start Zookeeper and expose port 2181 for use by the host machine
docker run -d --name zookeeper -p 2181:2181 confluent/zookeeper

# Start Kafka and expose port 9092 for use by the host machine
docker run -d --name kafka -p 9092:9092 --link zookeeper:zookeeper confluent/kafka

# Start Schema Registry and expose port 8081 for use by the host machine
docker run -d --name schema-registry -p 8081:8081 --link zookeeper:zookeeper \
--link kafka:kafka confluent/schema-registry

# Start REST Proxy and expose port 8082 for use by the host machine
docker run -d --name rest-proxy -p 8082:8082 --link zookeeper:zookeeper \
--link kafka:kafka --link schema-registry:schema-registry confluent/rest-proxy

3) Everything now is ready to continue


Working with kafka
==================

Create the files producer.php and consumer.php as follows in this section. For testing, open 2 terminals and run first the consumer.php script. On the other terminal run the producer.php script many times as you want. You should see how the message from the producer is picked by the consumer.

PHP Producer (producer.php)
------------
<?php

$avro_schema = [
"namespace" => "yournamespace",
"type" => "record",
"name" => "machineLog",
"doc" => "That is the documentation",
"fields" => [
["name" => "host", "type" => "string"],
["name" => "log", "type" => "string"],
]
];

$records = [
[ "host" => "hostname1", "host" => "cpu 67590 0 28263 13941723 602 7 1161 0 0 0" ],
[ "host" => "hostname2", "host" => "cpu0 67591 0 28266 13944700 602 7 1161 0 0 0" ]
];

$msg = json_encode([
"value_schema" => $avro_schema,
"records" => $records
]);

//var_dump(json_encode($msg));
$kafka = new Kafka("192.168.99.100:9092");
try {
$kafka->produce("jsontest", $msg);
} catch (Exception $e) {
echo $e->getMessage() . PHP_EOL;
}

$kafka->disconnect(Kafka::MODE_PRODUCER);



PHP Consumer (consumer.php)
------------
<?php

$kafka = new Kafka("192.168.99.100:9092");
$partitions = $kafka->getPartitionsForTopic('jsontest');
$kafka->setPartition($partitions[0]);
$offset = 1;
$size = 1;

while (1) {
try {
$messages = $kafka->consume("jsontest", $offset, $size);
if (count($messages) > 0) {
foreach ($messages as $message) {
echo $message . PHP_EOL;
$offset += 1;
}
}
} catch (Exception $e) {
echo $e->getMessage() . PHP_EOL;
break;
}
}

$kafka->disconnect();


That's all! Enjoy Kafka!

How to run Laravel 5.3 in a 1and1 shared hosting

How to run Laravel 5.3 in a 1and1 shared hosting

1.- Go to your Control Panel and set the default PHP version to 5.6 for your current domain.

2.- Artisan script

Open artisan file and replace the first line to:

#!/usr/local/bin/php5.5

which corresponds to the 1and1 php5.5 path. Now you can run artisan using ./artisan.

3.- composer.json

Open composer.json file and replace all the php references to php5.5. For example:

[...]

"scripts": {
        "post-root-package-install": [
            "/usr/local/bin/php5.5 -r \"copy('.env.example', '.env');\""
        ],
        "post-create-project-cmd": [
            "/usr/local/bin/php5.5 artisan key:generate"
        ],
        "post-install-cmd": [
            "Illuminate\\Foundation\\ComposerScripts::postInstall",
            "/usr/local/bin/php5.5 artisan optimize"
        ],
        "post-update-cmd": [
            "Illuminate\\Foundation\\ComposerScripts::postUpdate",
            "/usr/local/bin/php5.5 artisan optimize"
        ]
    },

[...]

Now you can update your dependencies:

Run the command:

curl -sS https://getcomposer.org/installer | php5.5 

to get your composer.phar script. Now you can update as follow:

php5.5 composer.phar update


4) .htaccess

Go to your public folder and replace the default setup with the following:

5) .env

Of course you have to configure your database credentials an the rest of setups editing the file .env.

Happy Laravel

How to download a file grom S3 with Laravel

Today let's explain how to download a file from S3 using laravel. Next time I will explain how to upload the file.

1) Set up the bucket name. Ex. YOUR_BUCKET_NAME

2) Set up the bucket policy

{
  "Id": "PolicyXXXXXX",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "XXXXXX",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::testfcce/*",
      "Principal": "*"
    }
  ]
}
You can use the AWS policy generator for that:
http://awspolicygen.s3.amazonaws.com/policygen.html

3) Generate a credentials to have access to the bucket (if you don't have)

4) Change the config/filesystems.php to

        's3' => [
            'driver' => 's3',
            'key'    => env('S3_KEY'),
            'secret' => env('S3_SECRET'),
            'region' => env('S3_REGION'),
            'bucket' => env('S3_BUCKET'),
        ],

5) Add the keys to the .env file

S3_USER=YOUR_USER_NAME
S3_KEY=YOUR_KEY
S3_SECRET=YOUR_SECRET
S3_REGION=eu-central-1
S3_BUCKET=YOUR_BUCKET_NAME

6) Example how to check if a file called image001.jpg is in the bucket and in positive case get the URL

$s3 = Storage::disk('s3');

if ($s3->exists('image001.jpg'))
{
    $bucket = 'YOUR_BUCKET_NAME';

    return $s3->getDriver()->getAdapter()->getClient()->getObjectUrl($bucket, 'image001.jpg');
}


References URLs:

https://return-true.com/uploading-directly-to-amazon-s3-from-your-laravel-5-application/
http://www.server180.com/2016/01/upload-files-to-s3-using-laravel-52.html
https://chrisblackwell.me/upload-files-to-aws-s3-using-laravel/

Best

Setting up Capistrano for Laravel 5 in Ubuntu

Capistrano for Laravel 5 in Ubuntu

Install Capistrano in Ubuntu

gem install capistrano

SSH access

Capistrano deploys using SSH. Thus, you must be able to SSH from the deployment system to the destination system for Capistrano to work. You can test the SSH connection using a ssh client, e.g.

ssh your_username@destinationserver

If you cannot connect, in order to be able to do this you need two things:
1) An account on the target machine
2) Your public key added to ~/.ssh/authorized_keys file.

So, SSH into the staging machine and add your key to username’s ~/.ssh/authorized_keys. You should now be able to pull from git on the server without it asking you for identification.

Server privileges

In order to deploy, the user needs some special privileges.

First add a group named deploy to the server:

sudo groupadd deploy

We now need to add ourselves and any other users that need access (other developers) to the deploy group, and the user that executes our web server. Opening the /etc/group file, and look for the line that beings with deploy. We append users, comma delimited. In this example I am using a nginx server.

deploy:x:1003:your_username,nginx

Now we have a deploy group, whose members include us, any other developers who will be deploying and the web server user.

Add the deploy group to the /etc/sudoers. For that create file /etc/sudoers.d/01deply to enable members of deploy group to access root via sudo without passwords

%deploy ALL = (ALL) NOPASSWD: ALL

and set up 440 permissions to the file

chmod 440 /etc/sudoers.d/01wheel

Project structure

Now define the new structure for your project

projectname
 |-- components
 |-- deploy

all of then will have 755 permissions and g+i flag that indicates that old the subdirectories will have the same privileges as the project directory.

sudo mkdir -p new-project/components 
sudo mkdir -p new-project/deploy 
sudo chmod -R 775 new-project 
sudo chmod -R g+i new-project 
sudo chown -R your_username new-project 
sudo chgrp -R deploy new-project

On the local machine

Go to the Laravel root project's folder and run

cap install

The command creates the following structure:

mkdir -p config/deploy 
create config/deploy.rb 
create config/deploy/staging.rb 
create config/deploy/production.rb 
mkdir -p lib/capistrano/tasks 
create Capfile Capified

where staging.rb and deploy.rb are the files that we are going to work.
We have to create an additional file to store our credentials:

vi config/myconfig.rb

set :ssh_options, { user: 'your_username' }
set :tmp_dir, '/home/your_username/tmp'




Adding Task to Capistrano

Vendor files and config files

Edit the config file deploy.rb

vi config/deploy.rb

and add the next lines after the lock line.

set :application, 'new-project'
set :repo_url, 'git@github.com:githubusername/project.git'

set :deploy_to, '/var/www/new-project/deploy'

components_dir = '/var/www/new-project/components'
set :components_dir, components_dir

# Devops commands
namespace :ops do

  desc 'Copy non-git files to servers.'
  task :put_components do
    on roles(:app), in: :sequence, wait: 1 do
      system("tar czf .build/vendor.tar.gz ./vendor")
      upload! '.build/vendor.tar.gz', "#{components_dir}", :recursive => true
      execute "cd #{components_dir}
      tar -zxf /var/www/new-project/components/vendor.tar.gz"
    end
  end

end


Now configure the staging.rb condig file

vi config/deploy/staging.rb

adding the next lines:

role :app, %w{your_username@destinationserver}

require './config/myconfig.rb'

namespace :deploy do

  desc 'Get stuff ready prior to symlinking'
  task :compile_assets do
    on roles(:app), in: :sequence, wait: 1 do
      execute "cp #{deploy_to}/../components/.env.staging.php #{release_path}"
      execute "cp -r #{deploy_to}/../components/vendor #{release_path}"
    end
  end

  after :updated, :compile_assets

end


# Devops commands
namespace :ops do

  desc 'Copy non-git ENV specific files to servers.'
  task :put_env_components do
    on roles(:app), in: :sequence, wait: 1 do
      upload! './.env.staging.php', "#{deploy_to}/../components/.env.staging.php"
    end
  end

end

Deploy

Let's try if everything works. Check if the tasks are available for cap writing cat -T.

Now let's transfer the config and vendor files to the server:

cap staging ops:put_components 
cap staging ops:put_env_components

Once the files are in the server we can deploy

cap staging deploy



Possible errors

Is you receive an error like this:

the deploy has failed with an error: Exception while executing as your_username@destinationserver: git exit status: 128 git 
stdout: Nothing written git stderr: Error reading response length from authentication socket. 
Permission denied (publickey). 
fatal: Could not read from remote repository.

that means you have to follow an additional step in your local terminal. Run

eval $(ssh-agent) 
ssh-add ~/.ssh/id_rsa

And try again, e.g.

cap staging deploy

That's all

Hope helps

Rustic way to import your AngularJS configuration from an external JS file

Many times we want to store our constants/config details into a different file to simplify the deployment process or simply to organise the code.

Here you can see a very basic way to save your app's config file into a different location.

https://jsfiddle.net/sergioloppe/bvajr7bw/

The code is self-explanatory.

Hope helps

How to avoid errors using the function eval in php (under any framework)

Sometimes when you have to evaluate some mathematical expressions with eval it is difficult to control the errors even using try-catch. Here you have a trick to solve that problem:


$calculation = '1/0';

$result = @eval($calculation . "; return true;");
if($result) {
$rule2_val = doubleval(@eval($calculation));
}
else {
return 'Impossible to perform the calculation';
}

Hope helps!

Using Illuminate/Html in Laravel 5.2

As you know, long time ago, Illuminate/Html was remove from laravel because it is not a core feature. Today I need to use Form for one of my projects and I am going to show how to include in Laravel 5.2. (I prefer not to use Form but you never know).

1) Add to composer.json

"illuminate/html": "5.*"

2) Update your project

composer update

3) Edit the file config/app.php and add to providers

Illuminate\Html\HtmlServiceProvider::class,

4) Edit config/app.php and add to aliases

        'Html'      => Illuminate\Html\HtmlFacade::class,
        'Form'      => Illuminate\Html\FormFacade::class,

5) Check it for example using the code:

{!! Form::open() !!}

{!! Form::text('name', @$name) !!}

{!! Form::password('password') !!}

{!! Form::submit('Send') !!}

{!! Form::close() !!}


If you receive the error:

[Symfony\Component\Debug\Exception\FatalErrorException]                   

  Call to undefined method Illuminate\Foundation\Application::bindShared() 

that means you need an additional step. Laravel 5.2 use singleton instead bindShared which is deprecated. You have to change that in your vendor HtmlServiceProvider as:

/**

* Register the HTML builder instance.

*

* @return void

*/

protected function registerHtmlBuilder()

{

$this->app->singleton('html', function($app)

{

return new HtmlBuilder($app['url']);

});

}

/**

* Register the form builder instance.

*

* @return void

*/

protected function registerFormBuilder()

{

$this->app->singleton('form', function($app)

{

$form = new FormBuilder($app['html'], $app['url'], $app['session.store']->getToken());

return $form->setSessionStore($app['session.store']);

});

}

Hope it helps

Github Two-Factor Authentication on the command line

Today is the day to deploy things and... also the day to set up the Github's two-factor authentication on the command line. In order to be able to work with your repositories once you set up your w-factor authentication follow the next steps:

1. Log into your Github account and go to Settings
2. Select the "Personal Access Tokens" tab on the left-hand navigation
3. Click on "Generate new" personal access token and enter a proper description
4. Copy the 40 characters long token and use it as your password on the command line

That's all. 

Hope help!

Installing Ruby on Rails in a Ubuntu/trusty64 Vagrant Box under OS El Capitan

To install Ruby on Rails in a Ubuntu Vagrant Box in a Mac follow the steps below:

1) Download and install vagrant and virtualbox
  - http://www.vagrantup.com/downloads
  - https://www.virtualbox.org/wiki/Downloads

2) Install the Ubuntu box
# vagrant init ubuntu/trusty64

4) Set up the forwarding

# vi Vagrantfile

Add the line in the forwarded port mapping

config.vm.network "forwarded_port", guest: 3000, host: 3030, protocol: 'tcp', auto_correct: true

5) Runing Ubuntu

# vagrant up --provider virtualbox
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 3000 (guest) => 3030 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default: 
    default: Guest Additions Version: 4.3.34
    default: VirtualBox Version: 5.0
==> default: Mounting shared folders...
    default: /vagrant => /Users/sergio/vagrant_boxes
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.


6) Access the virtual machine
# vagrant ssh

Now it's time to install Ruby on Rails. Follow the next steps_
1) Installing dependencies

# cd
# sudo apt-get update
# sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

2) Installing Ruby dev packages

# sudo apt-get install ruby-all-dev
# sudu updatedb

3) Installing rbenv

# cd
# git clone git://github.com/sstephenson/rbenv.git .rbenv
# echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
# echo 'eval "$(rbenv init -)"' >> ~/.bashrc
# exec $SHELL

# git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
# echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
# exec $SHELL

# git clone https://github.com/sstephenson/rbenv-gem-rehash.git ~/.rbenv/plugins/rbenv-gem-rehash

# rbenv install 2.2.3
# rbenv global 2.2.3
# ruby -v

4) If you want to avoid installing the packages documentation execute the following commands

# echo "gem: --no-ri --no-rdoc" > ~/.gemrc
# gem install bundler

4) Installing NodeJS

# curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
# sudo apt-get install -y nodejs

5) Installing Rails

# gem install rails -v 4.2.4
# rails -v

6) Set up PostgreSQL

# sudo sh -c "echo 'deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main' > /etc/apt/sources.list.d/pgdg.list"

# wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -

# sudo apt-get update
# sudo apt-get install postgresql-common
# sudo apt-get install postgresql-9.3 libpq-dev

7) Create a PostgrSQL user

# sudo -u postgres createuser INSERT_THE_USERNAME_HERE -s
# sudo -u postgres psql
postgres=# \password INSERT_THE_USERNAME_HERE

Now it is time to check the installation creating a rails application.

1) We will use sqlite in this case

# rails new myapp

If you want to use your PostgreSQL server use instead

# rails new myapp -d postgresql 

2) Go to the new app and create the database

# cd myapp
# rake db:create

3) Run the rails server

# rails s -b 0.0.0.0
=> Booting WEBrick
=> Rails 4.2.4 application starting in development on http://0.0.0.0:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
[2016-01-03 16:53:10] INFO  WEBrick 1.3.1
[2016-01-03 16:53:10] INFO  ruby 1.9.3 (2013-11-22) [x86_64-linux]
[2016-01-03 16:53:10] INFO  WEBrick::HTTPServer#start: pid=1956 port=3000


4) Check the server from your local machine at port 3030: http://127.0.0.1:3030

Hope help!